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Abstract 

r  <r 

This  investirgabien'  examined  the  effects  a  reliability 
improvement  warranty  (RIW)  had  on  the  actual  operational 
reliability  of  the  warranted  avionics  equipment.  To  accom¬ 
plish  this,  the  most  comprehensive  DOD  application  of  RIW  to 
date,  the  F-16  RIW,  was  investigated.  This  study  was  de¬ 
signed  to  answer  the  question  of  whether  or  not  a  warranted  -> 
system  was  significantly  more  reliable  than  it  would  have 
been  without  the  warranty.  Specifically,  the  observed  mean 
flight  hours  between  failures  (MFHBFs )  and  the  observed  re¬ 
liability  growths  of  the  warranted  F-16  equipment  were  com¬ 
pared  to  those  of  functionally  similar  non-warranted  F-15 
equipment.  Also,  the  reliability  growth  of  the  F-16  war¬ 
ranted  equipment  was  compared  to  that  of  other  non-warranted 
F-16  equipment.  The  AMSAA  Reliability  Growth  Model  was  used 
as  a  basis  for  the  reliability  growth  analyses.  Comparable 
life  cycle  time  periods  for  each  aircraft  were  studied,  using 
AFM  66-1  D056  failure  and  flight  time  data. 

The  results  of  the  investigation  indicated  that,  in 
general,  the  observed  MFHBFs  of  the  F-16  warranted  equipment 
were  statistically  greater  than  the  MFHBFs  of  the  equivalent 
F-15  equipment.  The  same  could  not  be  concluded  for  the 


reliability  growths  of  the  F-16  and  equivalent  F-1 5  equip¬ 
ment.  However,  the  reliability  growth  of  the  warranted  F-16 
equipment  was  found  to  be  statistically  greater  than  the 
reliability  growth  of  the  non-warranted  F-16  equipment.  The 
study  concluded  that  the  observed  increased  reliability  and 
reliability  growth  rate  of  the  warranted  F-16  equipment,  was 
due,  at  least  in  part,  to  the  F-16  RIW.  -  -  -  . 
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A  COMPARATIVE  EVALUATION  OF  THE  RELIABILITY  IMPROVEMENT 
IN  LINE  REPLACEABLE  UNITS  WARRANTED  UNDER  THE 
F-16  RELIABILITY  IMPROVEMENT  WARRANTY 

I .  Introduction 

Overview 

Weapon  system  quality,  reliability,  and  availability  are 
obviously  major  concerns  of  the  Air  Force,  especially  since 
our  systems  have  become  extremely  complex  in  recent  years. 
Within  just  the  past  three  years,  reliability  and  support- 
ability  have  become  very  important  issues  in  the  Department 
of  Defense  (DOD)  system  acquisition  process.  An  increase  in 
system  reliability,  along  with  the  corresponding  decrease  in 
supportability  costs,  can  be  viewed  as  the  primary  factors 
responsible  for  lowering  operating  and  support  costs,  the 
largest  portion  of  the  system's  total  life  cycle  cost.  Reli¬ 
ability  and  supportability  were  two  of  the  signiticant  issues 
addressed  in  the  "consolidated"  Defense  Acquisition  Improve¬ 
ment  Program  initiatives  introduced  in  1983  by  the  Deputy 
Secretary  of  Defense,  Paul  Thayer  (1:11;  2:9).  This  high 
level  concern  for  increased  reliability  and  supportability, 
along  with  the  ever  increasing  spending  constraints  put  on 
the  Air  Force  by  Congress  (3),  are  two  of  the  driving  factors 
that  have  influenced  DOD  to  explore  contract  reliability  and 
supportability  assurances  for  use  in  the  acquisition  process. 


Probably  the  most  prevalent  among  these  assurances  is  the 
warranty,  and  more  specifically,  the  reliability  improve¬ 
ment  warranty. 

In  late  1983,  the  interest  in  warranties  was  elevated  to 
the  congressional  and  presidential  levels  with  the  passage  of 
Section  794  of  the  1984  DOD  Appropriation  Act  (2:9;  4:14; 
5:44).  This  bill  required  that  performance  warranties  be 
procured  for  all  weapon  systems,  unless  waived  by  the  Secre¬ 
tary  of  Defense  for  national  security  reasons  or  if  the  re¬ 
quired  warranty  were  not  cost-effective  (2:9;  5:44).  Need¬ 
less  to  say,  this  high  level  congressional  interest  makes  the 
subject  of  warranties  a  major  concern  for  the  DOD. 

A  warranty  is  "a  promise  that  certain  facts  are  truly  as 
they  are  represented  to  be  and  that  they  will  remain  so, 
subject  to  any  specified  limitations"  (6:1423).  Through  a 
written  warranty,  the  supplier  promises  that  the  "material  or 
workmanship  is  defect  free  or  will  meet  a  specified  period  of 
time"  (6:1424).  One  of  the  purposes  of  the  warranty  is  to 
protect  the  buyer,  or  in  this  case,  the  government,  against 
defects  that  cannot  be  detected  prior  to  purchase  and  that 
only  become  apparent  after  the  system  has  been  used.  It  thus 
gives  the  government  an  increased  time  period  to  assert  its 
rights.  The  Federal  Acquisition  Regulation  (FAR)  defines  a 
warranty  as 

a  promise  or  affirmation  given  by  a  contractor  to 
the  Government  regarding  the  nature,  usefulness,  or 
condition  of  the  supplies  or  performance  of  ser¬ 
vices  furnished  under  the  contract.  The  general 
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purposes  of  a  warranty  in  a  Government  contract  are 
(1)  to  delineate  the  rights  and  obligations  of  the 
contractor  and  the  Government  for  defective  items 
and  services  and  (2)  to  foster  quality  performance 
(7:46-9). 

The  warranty  is  actually  a  subset  of  a  larger  class  of 
product  assurances,  called  product  performance  agreements. 

The  Product  Performance  Agreement  Guide,  developed  jointly  in 
1979  by  senior  Air  Force  and  industry  representatives,  lists 
23  different  types  of  product  performance  agreements,  eight 
of  which  are  directly  called  warranties,  in  one  form  or 
another  (8:Part  B).  In  general,  the  intent  of  product  per¬ 
formance  agreements  is  to  provide  an  incentive  for  the  con¬ 
tractor  to  enhance  both  the  performance  and  supportability  of 
his  product  "during  interim  contractor  support  before  transi¬ 
tioning  into  organic  support"  (9:26).  The  underlying  assump¬ 
tion  is  that  the  contractor  will  be  motivated  not  only  to 
develop  a  high  quality  product,  but  also  to  continually  im¬ 
prove  that  product's  reliability  and  supportability  through¬ 
out  its  functional  life.  The  bottom  line  is  that  this  in¬ 
creased  system  reliability  should  reduce  the  life-cycle  costs 
by  reducing  the  largest  part  of  the  total  cost  of  ownership, 
the  operating  and  support  (O&S)  costs. 

If  the  volume  of  literature  is  proportional  to  the 
frequency  of  use  or  dollars  involved,  then  ti e  reliability 
improvement  warranty  (RIW)  is  by  far  the  most  popular  product 
performance  agreement  used  by  the  DOD  in  the  recent  past. 

The  RIW  is  used  to  provide  an  incentive  to  the  contractor  to  " 


design  and  produce  systems  with  increased  reliability  and 
reduced  repair  costs  (10:3-1;  11:1).  For  this,  the  govern¬ 
ment  pays  an  agreed  upon  fixed  price,  while  the  contractor 
agrees  to  repair  or  replace,  if  required,  the  failed,  war¬ 
ranted  systems  returned  to  his  facility,  at  no  further  cost 
to  the  government.  The  RIW  contract  is  normally  awarded 
concurrently  with  the  production  contract  (10:3-1).  The 
contractor  is  highly  motivated  to  provide  a  highly  reliable 
and  supportable  system  because  of  the  fixed  price  nature  of 
the  RIW  contract  -  the  more  he  can  reduce  the  repair  costs, 
the  more  profit  he  can  realize  (11:3).  Under  this  concept, 
the  contractor  can  further  improve  system  reliability  or 
supportability  during  production  through  engineering  change 
proposals  (ECPs),  again  at  no  additional  expense  to  the 
government  (10:7-38;  11:3). 

In  its  pamphlet,  Interim  Guidelines  -  Reliability 
Improvement  Warranty  (RIW) ,  the  Air  Force  Directorate  of 
Procurement  Policy  estimates  that  most  increases  in  relia¬ 
bility  as  a  result  of  a  RIW  will  occur  in  the  initial  years 
after  the  system's  deployment  (11:4).  It  also  states  that 
"after  the  equipment's  reliability  and  maintainability  have 
been  satisfactorily  demonstrated  through  field  use",  the 
Government  may  evaluate  the  RIW's  effectiveness  (11:4).  With 
this  point  in  mind,  in  1983,  Parkinson  and  Schoolcraft  con¬ 
cluded  that  very  few  studies  have  ever  really  been  undertaken 
to  determine  RIW  effectiveness  for  DOD  applications  (12:5). 
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Problem  Statement 


During  the  mid  to  late  1970s  and  early  1980s,  DOD  pro¬ 
cured  numerous  weapon  system  components  using  the  RIW  con¬ 
cept.  A  review  of  the  literature  indicated  that  apparently 
very  few  RIW  programs  have  ever  been  assessed  for  their  cost 
effectiveness  (through  reliability  improvement)  at  the  com¬ 
pletion  of  the  warranty.  Very  few  research  studies  have  ever 
evaluated  the  RIW  in  terms  of  whether  or  not  it  actually 
caused  an  increase  in  a  warranted  system's  reliability. 

Since  this  acquisition  approach  has  the  potential  for  achiev¬ 
ing  increased  reliability  as  well  as  economical  supporta- 
bility,  and  since  warranties,  in  general,  are  now  subjects  of 
high  level  congressional  and  DOD  interest,  there  is  an  impor¬ 
tant  need  to  evaluate  the  effectiveness  of  RIW. . 

Background 

Introduction.  The  DOD  major  weapon  system  acquisition 
process  in  the  past  typically  emphasized  initial  acquisition 
cost,  schedule,  and  system  performance,  but  not  reliability 
and  supportability .  Because  of  this  lack  of  emphasis,  "past 
failures  to  meet  reliability  goals  have  resulted  in  decreased 
performance  capability,  increased  maintenance  costs,  more 
expensive  spare  parts  procurements,  and  costly  product  im¬ 
provement  efforts"  (13:20).  Only  within  the  past  ten  years, 
with  the  advent  of  integrated  logistics  support,  has  design 
reliability  and  supportability  been  given  equal  consideration 
with  cost,  schedule,  and  performance.  But  how  does  the  DOD 


motivate  the  contractor  to  design  reliability  and  supporta- 
bility  into  the  system,  especially  when  it  is  to  the  contrac¬ 
tor's  advantage  to  deliver  the  lowest  design  reliability  the 
DOD  will  accept?  For,  by  increasing  system  reliability  the 
contractor  increases  "his  manufacturing  and  quality-control 
costs"  and  reduces  "his  opportunity  for  sale  of  spare  units, 
service  contracts,  and  replacement  parts  to  provide  opera¬ 
tional  support"  (14:2-2). 

This  need  to  provide  a  contractual  incentive  to  improve 

reliability  led  to  a  new  type  of  warranty  contract  -  the 

reliability  improvement  warranty  (RIW).  A  RIW  is  defined  as 

a  provision  in  either  a  fixed  price  acquisition,  or 
fixed  price  equipment  overhaul  contract  in  which: 

(a)  the  contractor  is  provided  with  a  monetary 
incentive,  throughout  the  period  of  the  warranty, 
to  improve  the  production  design  and  engineering  of 
the  equipment  so  as  to  enhance  the  field/opera¬ 
tional  reliability  and  maintainability  of  the 
system/equipment;  and  (b)  the  contractor  agrees 
that,  during  a  specified  or  measured  period  of  use, 
he  will  repair  or  replace  (within  a  specified  turn¬ 
around  time)  all  equipment  that  fails  (subject  to 
specified  exclusions,  if  applicable)  (11:5,6). 

The  RIW  is  thus  a  fixed  price  contractual  provision  which 
motivates  contractors  to  both  design  and  produce  systems  with 
inherently  low  failure  rates  and  repair  costs,  and  improve 
system  reliability  even  after  the  system  has  been  deployed. 
The  primary  objective  of  the  RIW,  then,  is  to  achieve  an 
increase  in  reliability,  even  though  the  major  expense  of  the 
RIW  contract  covers  the  anticipated  repair  services  (14:2-5). 
Before  the  advent  of  the  RIW,  the  risks  associated  with  de¬ 
ploying  state-of-the-art  technology  rested  squarely  on  the 


shoulders  of  the  Air  Force.  Poor  reliability  brought  on 
"high  O&S  costs  or  reduced  asset  availability"  (10:3-2). 

With  RIW,  however,  the  risks  are  shared  by  both  the  Air  Force 
and  the  contractor. 

The  mean  time  between  failure  (MTBF)  guarantee  is  often 
used  in  conjunction  with  a  RIW  (9:26;  10:3-2;  15:20).  Again, 
it  provides  an  incentive  for  the  contractor  to  increase  reli¬ 
ability  and  reduce  support  costs.  The  MTBF  guarantee  re¬ 
quires  the  contractor  to  institute  some  kind  of  corrective 
action  (modifications,  engineering  analyses,  ECPs)  if  the 
warranted  systems  fail  to  meet  the  contractually  specified 
MTBF  (16:7,8).  It  also  requires  the  contractor  to  provide 
the  government  additional  spares  to  support  its  normal  opera¬ 
tions,  as  outlined  in  the  contract  (10:4-2;  16:8). 

As  stated  earlier,  under  a  RIW  the  contractor  should  be 
highly  motivated  to  provide  a  highly  reliable  and  supportable 
system  because  of  the  fixed  price  nature  of  the  contract  - 
the  more  the  contractor  can  reduce  equipment  failures  and 
their  accompanying  repair  costs,  tha  more  profit  he  can 
realize.  "He  has  a  strong  incentive  to  achieve  or  exceed  the 
reliability  level  on  which  the  warranty  price  was  based" 
(14:2-5).  There  is  also  a  strong  incentive  for  the  contrac¬ 
tor  to  further  improve  system  reliability  through  no-cost 
ECPs  and  production  changes,  especially  during  the  initial 
stages  of  the  warranty  (10:7-38;  14:2-5).  Obviously,  since 
profit  is  a  strong  motivator  for  the  contractor,  it  is 


unlikely  that  he  would  introduce  no-cost  ECPs  during  the 
latter  stages  of  the  warranty  period,  especially  if  the 
expected  savings  in  repair  costs  is  less  than  the  costs  he 
would  incur  to  make  the  change.  In  fact,  in  an  Air  Command 
and  Staff  College  research  report  on  contractor  RIW  incen¬ 
tives  and  risks,  Major  Raymond  Hudkins  concluded  that  no  con¬ 
tractor  saw  any  real  incentive  to  introduce  RIW  ECPs  "after 
two  years  into  a  five  year  RIW"  (17:98).  Perhaps  the  real 
incentive  for  the  contractor  to  improve  design  reliability 
and  thereby  reduce  supportability  costs  starts  by  "stating  in 
the  development  contract  that  a  warranty  will  be  a  require¬ 
ment  in  the  production  contract"  (18:2). 

Before  discussing  the  RIW  in  detail,  a  review  of  the 
concepts  of  reliability  and  reliability  growth  is  in  order. 

Reliability  and  Reliability  Growth.  Certainly  the 
importance  of  weapon  system  reliability  can  never  be  over¬ 
stated;  it  provides  the  probability  that  the  system  will 
successfully  carry  out  its  mission.  From  an  economics  view¬ 
point,  the  higher  the  reliability,  the  greater  the  likelihood 
that  major  cost  savings  can  be  realized  over  the  life  of  the 
system.  Typically,  a  system's  life  cycle  is  made  up  of  three 
distinct  failure  phases:  an  early  failure  period,  a  random 
failure  period,  and  a  wear-out  failure  period  (19:8).  During 
the  early  failure  or  burn-in  period,  failures  normally  occur 
because  of  faulty  designs,  faulty  manufacturing,  or  inspec¬ 
tion  deficiencies  (19:8).  It  is  during  this  early  phase  that 


design  improvements  are  normally  made,  "product  tolerances 
are  reduced,"  and  manufacturing  operations  are  streamlined 
(20:11).  During  the  second  phase,  the  random  failure  period, 
"systems  experience  chance  failures  according  to  one  of  the 
several  probability  distributions,  normally  the  exponential" 
(20:11).  The  last  phase,  or  wear-out  period,  is  character¬ 
ized  by  an  increasing  failure  rate  due  to  the  deterioration 
of  the  system  components. 

Reliability  is  directly  related  to  the  life  cycle  cost 
of  a  weapon  system.  System  reliability  improvements  incorpo¬ 
rated  during  the  design  stage  cost  the  government  consider¬ 
ably  less  than  reliability  modifications  made  during  later 
stages  in  the  life  of  the  weapon  system.  Thus,  "it  is  desir¬ 
able  to  have  intensive  research  and  reliability  engineering 
in  the  design  phases,  rather  than  after  a  weapon  system  is  in 
the  operational  inventory"  (20:12).  Realistically,  for  one 
reason  or  another,  this  intensive  infusion  of  reliability 
into  the  design  phase  has  not  occurred.  Therefore,  nearly 
all  weapon  systems  experience  a  reliability  improvement  phase 
known  as  reliability  growth. 

Reliability  growth  is  the  "positive  improvement  of  the 
reliability  of  equipment  through  the  systematic  and  permanent 
removal  of  failure  mechanisms"  (21:3).  It  is  not  a  naturally 
occuring  characteristic  of  electronic  equipment  (22:12);  it 
requires  a  conscious  engineering  effort  to  attain  higher 
levels  of  reliability  through  the  elimination  of  design 


weaknesses.  Reliability  growth  can  take  place  at  any  point 
in  the  system’s  life  cycle.  Again,  the  earlier  in  the  life 
cycle  it  occurs,  the  less  it  will  cost. 

Reliability  growth  can  occur  in  one  or  more  of  the 
following  three  ways  (22:6): 

1.  Through  the  operation  of  the  equipment,  defective 
components  or  manufacturing  faults  can  be  exposed  and  re¬ 
placed. 

2.  Through  familiarization,  the  operators  become  more 
skilled  at  operating  the  equipment. 

3.  The  discovery  of  "errors  or  weaknesses  in  design, 
manufacturing,  or  related  procedures"  lead  to  their  correc¬ 
tion.  In  this  case  the  failure  mechanism  is  permanently 
removed . 

The  concept  of  reliability  growth  plays  a  very  important 
role  in  the  RIW  decision  process.  The  cost  effectiveness  of 
any  RIW  is  a  function  of  the  reliability  growth  expected  to 
take  place  over  the  warranty  period  (23:235),  in  addition  to 
the  incentive  the  RIW  provides  for  a  reliable  design. 

RIW  Historical  Background.  The  use  of  warranties  in  de¬ 
fense  contracts  is  not  new.  The  Armed  Services  Procurement 
Regulation  (ASPR),  in  1964,  contained  specific  guidelines  for 
the  use  of  warranties  in  firm-fixed-price  contracts.  As  in¬ 
structed  by  the  ASPR,  warranties  were  only  to  be  used  when 
the  warranty  protection  afforded  the  government  was  greater 
than  the  cost  of  the  warranty  (24:4).  At  that  time  DOD  used 
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warranties  mostly  for  "the  procurement  of  small,  off-the- 
shelf  items"  (25:2).  Then  in  1967,  a  "correction  of  defi¬ 
ciency"  clause  was  added  to  the  ASPR,  which  essentially 
attempted  to  hold  the  contractor  responsible  for  the  design 
and  quality  of  his  product.  It  required  the  contractor  to 
"correct  or  repair  defects  discovered  in  items  delivered  to 
the  government  under  that  contract"  (25:10).  The  Navy  was 
really  the  first  service  to  apply  the  warranty  on  a  signif¬ 
icant  contract.  It  used  a  failure  free  warranty  (FFW)  type 
of  contract  for  the  overhaul  of  gyros.  The  FFW  was,  in 
effect,  the  forerunner  of  the  RIW  (24:3).  The  Air  Force 
followed  suit  in  1969,  with  the  award  to  Lear  Siegler  of  a 
contract  for  the  acquisition  of  attitude  and  heading  gyros 
which  contained  a  FFW  (24:4).  For  the  next  ten  years  the 
services  awarded  very  few  additional  major  contracts  contain¬ 
ing  warranty  clauses.  During  the  mid-1970s,  however,  numer¬ 
ous  warranty  studies  gave  rise  to  the  reliability  improvement 
warranty  concept,  and  the  RIW  became  a  high  level  issue  in 
DOD.  With  the  publishing  of  the  DOD  pamphlet  containing  RIW 
guidelines  in  1975  ( Interim  Guidelines  -  Reliability 
Improvement  Warranty  (RIW) ) ,  the  military  services  were  "to 
undertake  a  trial  use  of  RIWs  in  a  number  of  electronic 
system/equipment  programs"  (24:5,6). 

Since  1975,  the  use  of  warranties  (especially  RIWs)  in 
Air  Force  acquisition  programs  has  increased  significantly. 
For  instance,  in  1979  there  were  44  programs  with  some  sort 


of  warranty  or  incentive  clauses  in  their  contracts  (24:7), 
and  according  to  the  Deputy  Under  Secretary  of  Defense  for 
Acquisition  Management,  Mary  Ann  Gilleece,  in  1979  "one-third 
of  our  4.1  million  types  of  items  in  inventory  were  covered 
by  some  form  of  warranty"  (9:26).  During  the  late  1970s, 

RIWs  were  applied  to  the  following  Air  Force  production  con¬ 
tracts:  the  ARN-118(V)  TACAN,  the  C-141/KC-135  inertial  nav¬ 

igation  system,  the  C-141  heading  system,  an  F-111  gyro  sys¬ 
tem,  and  nine  components  of  the  F-16  (17:2).  And  these 
represent  only  a  few  of  the  RXW  contracts  used  at  that  time. 
Of  these,  the  F-16  RIW  program,  the  focus  of  this  thesis,  was 
the  "most  comprehensive  and  complex  application  of  RIW  ever 
attempted"  within  DOD  (10:1-1).  The  RIW  contract  was  between 
the  Air  Force  and  General  Dynamics,  but  General  Dynamics,  in 
turn,  applied  the  contract  to  four  of  its  major  avionics  sub¬ 
contractors  (26:103).  This  leads  to  the  present,  where  the 
interest  in  warranties  has  been  elevated  to  the  congressional 
and  presidential  levels. 

In  late  1983,  Congress  passed  the  1984  DOD  Appropriation 
Act.  Section  794  of  that  Act,  authored  by  Senator  Mark 
Andrews  of  North  Dakota,  required  performance  warranties  for 
all  DOD  weapon  systems  (2:9).  Waivers  may  be  provided  by  DOD 
for  national  security  reasons  or  if  the  required  warranty 
were  not  cost-effective;  however,  such  a  waiver  must  be  re¬ 
ported,  in  writing,  to  four  separate  House  and  Senate  commit¬ 
tees  (5:44).  After  much  debate  and  a  lot  of  pressure  from 


the  defense  community,  several  revisions  were  made  to  the 
law,  giving  DOD  more  flexibility  in  using  warranties;  they 
are  included  in  the  recently  signed  1985  Defense  Authoriza¬ 
tion  Bill  (27:24). 

RIW  Advantages  Versus  Disadvantages .  As  might  be  ex¬ 
pected,  warranties  have  benefits  as  well  as  negative  aspects. 
The  government  and  the  contractor  are  not  necessarily  at 
opposing  ends  of  the  issue,  though.  What  is  an  advantage  for 
the  government  may  not  always  be  a  disadvantage  for  the  con¬ 
tractor.  While  the  literature  refers  to  numerous  RIW  advan¬ 
tages  and  disadvantages,  this  review  included  only  the  most 
relevant  and  the  most  often  cited  aspects.  The  majority  of 
these  positive  and  negative  aspects  should  be  classified  as 
perceived  or  intuitively  obvious,  since  it  appears  from  a 
review  of  the  literature  that  very  little  empirical  research 
has  been  conducted  on  the  actual  effects  of  the  RIW  in  gov¬ 
ernment  contracts. 

The  following  advantages  the  government  can  expect  actu¬ 
ally  deal  with  one  aspect  -  the  reduction  of  system  life- 
cycle  costs.  Under  a  RIW,  the  program  risks  as  well  as  the 
rewards  are  now  shared  by  both  the  government  and  the  con¬ 
tractor;  the  government  has  more  time  to  uncover  design  or 
production  defects  not  readily  apparent  at  the  time  of  deliv¬ 
ery  (10:3-2).  A  second  government  benefit  is  that  the  con¬ 
tractor  has  the  incentive  to  provide  a  highly  reliable  system 
with  low  support  costs,  which  ultimately  results  in  reduced 


system  life-cycle  costs.  The  contractor  is  also  motivated  by 
a  potential  increase  in  profits  to  improve  the  system  relia¬ 
bility  and  supportability  after  production.  He  does  this 
through  design  improvements  at  no  cost  to  the  government 
(10:3-1,2;  25:2).  Another  benefit  to  the  government  is  that 
the  cost  of  ownership  during  the  warranty  period  is  fixed  to 
a  degree,  since  any  repairs  will  be  covered  by  the  warranty; 
this  reduces  the  need  for  future  program  dollars  for  repairs 
(25:18;  2:10).  One  last  important  benefit  is  that  design  and 
production  deficiencies  are  discovered  early  on  and  not  per¬ 
petuated  "into  on-going  production  or  into  new  systems  under 
development"  when  warranted  equipment  is  returned  to  the 
contractor  for  repair  (26:103). 

Probably  the  most  important  benefit  to  the  contractor  is 
the  possibility  of  increased  profits.  Very  simply,  if  the 
contractor's  actual  repair  or  replacement  costs  plus  his  ad¬ 
ministrative  costs  for  warranty  service  are  less  than  the 
amount  paid  by  the  government  for  the  warranty,  he  earns  a 
profit  (8:10).  Another  benefit  to  the  contractor  is  the  "op¬ 
portunity  to  receive  feedback  on  equipment  operation"  in  its 
operating  environment  (17:31).  This  knowledge  may  be  invalu¬ 
able  for  his  future  designs  or  even  more  importantly,  for 
design  or  production  changes  during  current  production  runs. 

The  potential  RIW  disadvantages  to  the  government  are 
numerous.  First  of  all,  acquisition  costs  will  certainly  in¬ 
crease  -  without  exception,  warranties  cost  money  (9:26). 


Additionally,  it  is  extremely  difficult  to  accurately  predict 
the  future  costs  and  benefits  that  are  expected  to  be  the  re¬ 
sult  of  a  warranty.  Because  of  this  limitation,  most  of  the 
costs  are  derived  from  nothing  more  than  educated  guesses. 
Another  disadvantage  is  an  increase  in  administrative  com¬ 
plexity.  The  RIW  administration  involves  added  record  keep¬ 
ing,  reporting,  and  increased  monitoring  (10:7-31).  Mary  Ann 
Gilleece  points  out  another  potential  RIW  disadvantage  which 
results  when  the  contactor  changes  a  system  defect  or  im¬ 
proves  a  system  supportability  aspect  under  the  warranty. 

The  problem  is  that,  unknown  to  the  contractor,  this  design 
change  could  possibly  require  resultant  changes  to  inter¬ 
facing  systems  such  as  support  equipment  (9:26).  So  while 
the  system  repair  costs  may  be  reduced  by  this  design  change, 
the  overall  costs  to  the  government  may  actually  increase  due 
to  expensive  changes  required  in  the  support  equipment. 
Another  problem  arises  when  a  warranted  system  fails.  Under 
almost  all  RIW  contracts  in  the  past.  Air  Force  maintenance 
personnel  were  only  able  to  do  fault  isolation  and  other 
similar  tests  on  failed  equipment.  They  had  neither  the 
"tools"  nor  the  expertise  to  repair  the  the  defective,  war¬ 
ranted  equipment;  they  could  only  replace  it  and  send  it  to 
the  contractor  (10:7-3).  This  type  of  maintenance  support 
concept  would  obviously  be  unacceptable  in  combat  situations. 
Also,  additional  training  would  be  required  for  maintenance 
personnel  to  preclude  them  from  voiding  the  warranty  through 


improper  maintenance  actions.  Especially  in  light  of  recent 
attempts  to  increase  competition  in  the  acquisition  process, 
one  last  disadvantage  is  that  the  requirement  to  have  a  war¬ 
ranty  included  in  a  contract  may  actually  be  detrimental  to 
small  businesses.  They  normally  have  only  limited  resources 
to  back  up  the  warranty  (5:44). 

Many  of  the  above  RIW  disadvantages  also  affect  the  con¬ 
tractor.  It  is  just  as  difficult  for  the  contractor  to  accu¬ 
rately  predict  the  future  costs,  failure  rates,  etc.,  as  it 
is  for  the  government.  The  shared-risk  advantage  to  the 
government  becomes  a  disadvantage  to  the  contractor.  "Mis¬ 
handling  or  improper  use"  of  warranted  equipment  by  Air  Force 
personnel  may  cause  failures  which  are  beyond  the  control  of 
the  contractor  (2:10). 

The  F-1 6  RIW  Program 

Contractual  Background  (28).  The  F-1 6  production  con¬ 
tract  ( F33657-75-C-031 0 )  contained  a  special  provision,  J.63, 
which  left  the  option  open  to  the  government  of  including  a 
RIW  and  a  RIW  with  a  MTBF  guarantee  in  the  final  contract. 

In  1975,  the  government  negotiated  with  General  Dynamics  to 
exercise  this  option  on  U.S.  F-1 6s  to  include  the  coverage  of 
12  line  replaceable  units  (LRUs).  Near  the  end  of  1975,  the 
European  Participating  Governments  (EPGs)  expressed  the  in¬ 
terest  to  exercise  the  RIW  option,  also.  (The  F-1 6  program 
is  a  multi-national  fighter  program  -  the  countries  involved 
are  Belgium,  Denmark,  The  Netherlands,  Norway,  and  the  U.S.) 


During  1976,  the  U.S.  conducted  economic  analyses  on  the  RIW 
covering  the  12  LRUs.  In  October  of  that  year,  negotiations 
were  reinitiated  with  General  Dynamics  to  extend  RIW  coverage 
to  the  four  EPGs  as  well.  The  initial  RIW  price  for  the  EPGs 
alone  was  determined  to  be  prohibitive,  so  the  Air  Staff 
approved  a  combined  U.S./EPG  RIW  program.  Negotiations  were 
completed  in  February  1977  with  the  signing  of  a  separate  RIW 
contract  (F33657-77-C-0062 )  for  the  coverage  of  nine  LRUs  for 
all  five  nations,  with  a  not-to-exceed  cost  of  $44  million. 
Later  that  year  General  Dynamics  submitted  a  def initization 
proposal  for  the  RIW  contract  which  identified  numerous  con¬ 
ditions  of  cost.  The  government  rejected  14  of  these  cost 
conditions,  which  resulted  in  further  negotiations  and  fact 
finding  during  1978  and  early  1979.  The  RIW  started  in  Jan¬ 
uary  1979,  even  though  the  contract  was  not  finalized.  In 
May  1979,  the  F-16  Contracting  Officer  issued  a  final  deci¬ 
sion  after  negotiations  were  discontinued.  In  September, 
General  Dynamics  appealled  the  decision  to  the  Armed  Services 
Board  of  Contract  Appeals  (ASBCA).  With  the  Air  Force  re¬ 
sponse  to  the  ASBCA  in  April  1980,  it  was  obvious  no  settle¬ 
ment  could  be  reached.  The  two  parties  would  go  to  court; 
pre-trial  hearings  began  in  September  of  that  year  in  Wash¬ 
ington,  D.C.  Before  the  case  was  settled  in  court,  the  deci¬ 
sion  was  made  on  20  March  1981  to  commence  out  of  court  set¬ 
tlement  negotiations.  General  Dynamics  and  the  Air  Force 
finally  reached  a  settlement  in  April.  The  Air  Force 


compromised  on  several  of  the  cost  conditions,  however  the 
not-to-exceed  cost  remained  at  $44  million. 

Program  Description.  The  F~16  program  included  nine  of 
the  original  twelve  LRUs  and  applied  to  the  first  250  USAF 
and  first  192  EPG  production  F-16s  delivered,  and  their  asso¬ 
ciated  spares  (10:4-1).  The  nine  LRUs  covered  and  their  man¬ 
ufacturers  included  the  following  (10:4-1;  26:103): 

APG-66  Radar  (five  major  LRUs)  -  Westinghouse 
Head-up  Display  (two  major  LRUs)  -  Marconi  Avionics 
Inertial  Navigation  Unit  -  Singer-Kearfott 
Flight  Control  Computer  -  Lear-Siegler 
The  radar  transmitter  and  the  head-up  display  electronic  unit 
were  also  covered  by  a  MTBF  guarantee.  The  warranty  period 
lasted  four  years,  from  January  1979  to  December  1982.  The 
warranty  actually  covered  a  four  year  period  or  a  combined 
total  of  300,000  flight  hours  on  the  entire  warranted  fleet 
of  250  U.S.  and  192  EPG  production  aircraft,  whichever  oc- 
cured  first  (10:1-3).  The  actual  total  flight  time,  how¬ 
ever,  only  reached  around  200,000  hours  in  four  years  (28). 

What  made  this  the  most  complex  application  of  RIW  was 
not  only  the  number  of  warranted  units  but  also  the  fact  that 
approximately  halfway  through  the  warranty  period  six  of  the 
LRUs  transitioned  to  a  RIW  at  the  shop  replaceable  unit  (SRU) 
level  (10:4-2).  At  the  LRU  level,  the  entire  defective  LRU 
assembly  is  sent  back  to  the  contractor  for  repair.  At  the 
SRU  level,  organic  Air  Force  maintenance  fault-isolates  down 


to  the  SRU  level  using  Avionics  Intermediate  Shop  (AIS)  test 
equipment,  and  then  only  the  defective  SRU  is  returned  to  the 
contractor  (10:4-2).  The  radar  antenna  started  the  warranty 
period  at  the  SRU  level,  while  the  two  LRUs  covered  by  the 
MTBF  guarantee  remained  at  the  LRU  level  throughout  the 
period  (10:4-2).  The  RIW  contract  specified  the  conversion 
to  the  SRU  level  would  be  completed  by  July  1980.  Due  to 
shortages  of  AIS  test  equipment,  technical  orders,  and  SRU 
spares,  the  actual  conversion  was  not  completed  until  January 
1981,  however  (28). 

Interim  Evaluation.  In  1979,  the  F-16  Program  Office 
contracted  with  ARINC  Research  Corporation  to  evaluate  the 
effectiveness  of  the  F-16  RIW  program  (28).  The  evaluation 
was  conducted  about  halfway  through  the  warranty  period,  with 
only  16  percent  of  the  projected  total  program  flying  hours 
flown  (29:ix).  The  following  is  the  summary  of  that 
evaluation: 

Based  on  an  interim  evaluation,  ARINC  Research 
concludes  that  the  F-16  RIW  program  has  been  bene¬ 
ficial  to  date.  The  government  and  all  the  con¬ 
tractors  will  most  likely  benefit  financially  from 
the  program.  Reliability  levels  of  both  warranted 
and  nonwarranted  equipments  are  acceptable  in  terms 
of  original  program  office  expectations;  no 
outstanding  reliability  gains  were  observed  for  the 
warranted  equipment  (29:ix). 

Research  Obi ective 

The  objective  of  this  study  was  to  determine  if  the  Air 
Force  appl. cation  of  RIW  in  the  past  has  been  effective;  has 
the  warranty  achieved  the  higher  reliability  and  reduced  cost 


potential  the  Air  Force  paid  for?  Was  the  warranted  equip¬ 
ment  significantly  more  reliable  than  it  would  have  been 
without  the  warranty?  In  order  to  accomplish  this,  the  F-16 
RIW  program,  which  has  been  the  most  comprehensive  applica¬ 
tion  of  RIW  to  date,  was  investigated.  ARINC  recommended  in 
its  interim  evaluation  that  a  follow-on  report  be  prepared  to 
more  precisely  define  the  economics  and  reliability  experi¬ 
ence  upon  completion  of  the  F-16  RIW  program.  This  research 
would,  in  part,  do  just  that,  by  evaluating  the  reliability 
growth  exhibited  by  the  nine  warranted  LRUs  during  the  entire 
warranty  period. 

Research  Questions 

1.  As  compared  with  the  reliabilities  ( MTBFs )  of  nine 
functionally  similar  non-warranted  LRUs  in  the  F-15,  are  the 
reliabilities  (MTBFs)  of  the  nine  F-16  warranted  LRUs 
significantly  higher? 

2.  As  compared  with  functionally  similar  non-warranted 


LRUs  in  both  the  F-15  and  F-16,  did  the  nine  F-16  warranted 
LRUs  exhibit  a  significantly  higher  reliability  growth  during 
the  first  four  years  of  operation? 


II.  Methodolo 


Overview 

Chapter  I  provided  a  background  of  and  the  justification 
for  the  research  on  the  effectiveness  of  the  F-16  reliability 
improvement  warranty  program.  The  basic  problem  and  the 
formulated  research  questions  were  defined.  This  chapter 
describes  in  detail  the  specific  procedures  employed  in  the 
research  design,  including  both  data  acquisition  and  data 
analysis.  Definitions  of  key  terms,  as  they  relate  to  this 
study,  are  presented  first.  Then,  a  detailed  discussion  of 
the  data  used  for  the  study,  followed  by  the  methodology  de¬ 
vised  to  answer  each  of  the  research  questions  is  presented. 
Finally,  the  assumptions  and  the  limitations  of  the  research 
method  are  summarized. 

As  stated  in  Chapter  I,  the  F-16  Program  Office  con¬ 
tracted  with  the  ARINC  Research  Corportation  to  evaluate  the 
effectiveness  of  the  F-16  RIW  at  the  halfway  point  of  the 
warranty  period.  The  research  questions  posed  by  this  study 
were  answered  by  analyzing  available  data  for  the  entire  four 
year  warranty  period,  using,  for  the  most  part,  the  same 
concepts  and  procedures  used  by  ARINC  in  their  interim  eval¬ 
uation.  This  study  was  designed  to  analyze  the  effects  of  a 
RIW  on  the  reliability  of  a  system.  Specifically,  does  a 
warranted  system  have  a  significantly  greater  mean  time 


between  failure  than  a  functionally  similiar  non-warranted 
system? 

To  determine  whether  a  RIW  does  influence  the  relia¬ 
bility  of  the  warranted  equipment,  this  research  first  com¬ 
pared  the  field  reliabilities  of  the  nine  warranted  F-16  LRUs 
with  those  of  functionally  similar  non-warranted  F-15  LRUs. 
Then  the  growths  in  reliability,  over  a  four  year  period,  of 
the  F-15  LRUs  plus  nine  non-warranted  F-16  LRUs  were  compared 
with  the  reliability  growths  exhibited  by  the  nine  warranted 
F-16  LRUs.  Specifically,  this  analysis  was  designed  to 
detect  whether  the  contractors  made  substantial  efforts  to 
improve  equipment  reliability  during  early  production. 

Key  Terms  Defined 

The  following  key  terms  are  defined  as  they  apply  to 
this  study.  While  most  have  relatively  common  connotations, 
they  must  be  refined  to  reflect  their  exact  applications  in 
this  study. 

Reliability  is  "the  probability  that  an  item  can  perform 
its  intended  function  for  a  specified  interval  under  stated 
conditions"  (30:8). 

Reliability  Growth  is  "the  positive  improvement  of  the 
reliability  of  equipment  through  the  systematic  and  permanent 
removal  of  failure  mechanisms"  (21:3).  It  requires  a  con¬ 
scious  engineering  effort  to  attain  higher  levels  of  relia¬ 
bility  through  the  elimination  of  design  weaknesses. 


Failure  is  an  "event  in  which  a  previously  acceptable 
item  does  not  perform  one  of  more  of  its  required  functions 
within  the  specified  limits  under  specified  conditions" 
(21:3).  An  item  which  is  found  to  be  broken  or  damaged  which 
would  "cause  failure  under  operational  conditions"  is  also  a 
failure  (21 : 3 ) . 

Mean  Time  Between  Failures  ( MTBF )  is  the  average  time 
interval  between  failures,  measured  in  hours.  It  is  consid¬ 
ered  to  be  a  primary  measure  of  reliability  for  repairable 
items.  For  the  purpose  of  this  research  design.  Mean  Flight 
Hours  Between  Failures  ( MFHBF )  will  be  used  exclusively. 

MFHBF  is  equal  to  the  total  flying  time  of  the  equipment  for 
a  specified  time  interval  divided  by  the  number  of  relevant 
failures  during  that  interval. 

Line  Replaceable  Unit  (LRU)  is  "the  first  level  of  dis¬ 
assembly  below  the  system  level  that  would  be  carried  as  a 
line  item  of  supply  at  base  level"  (10:3-4).  It  is  a  "black 
box"  item  which  is  usually  removed  from  the  aircraft  and 
replaced  as  a  single  unit. 

Shop  Replaceable  Unit  ( SRU )  is  a  component  of  a  LRU  that 
has  a  unique  stock  number  and  is  maintained  by  base  level 
supply  to  support  LRU  intermediate  level  repair  (31:627).  In 
contrast  to  the  LRU,  which  is  removed  and  replaced  on  the 
flightline,  SRUs  require  disassembly  of  the  LRU  in  the  shop. 

A  circuit  board  assembly  is  an  example  of  a  SRU. 
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Work  Unit  Code  ( WUC )  is  a  unique  identifying  code  as¬ 
signed  to  LRUs,  SRUs,  and  other  components.  It  is  a  combina¬ 
tion  of  five  alpha-numeric  characters.  "The  first  two  char¬ 
acters  identify  the  system,  the  next  two,  the  subsystem,  and 
the  fifth,  the  component"  (31:748).  Codes  are  listed  in  the 
appropriate  weapon  system  WUC  Manual,  i.e.  T.O.  1F-16A-06. 

Data  Collection 

Maintenance  Data  Collection  System.  The  Air  Force 
Maintenance  Data  Collection  System  (MDCS),  as  directed  by  Air 
Force  Manual  66-1 ,  is  designed  to  provide  useful  maintenance 
data  to  all  management  levels  within  the  Air  Force.  MDCS  was 
the  only  data  source  common  to  both  the  F-1 6  and  F-1 5  air¬ 
craft  for  the  time  periods  involved  in  the  study.  While  the 
contractually  required  semiannual  RIW  data  reports  contained 
extensive  failure  data  on  the  nine  warranted  F-1 6  LRUs,  there 
was  no  similar  data  source  for  the  non-warranted  F-1 5  LRUs. 

So  MDCS,  or  more  specifically,  the  D056  system  was  thus  the 
source  for  the  data  used  in  this  analysis.  The  specific  re¬ 
port  used  was  the  LOG-LOE( AR)71 70  report  ( LOG-MMO( AR ) 71 70 
prior  to  1981),  which  was  formerly  called  the  "6-LOG  report". 
It  is  entitled  "Maintenance  Actions,  Man-Hours,  and  Aborts  by 
Work  Unit  Code"  (32:1).  The  data,  on  microfiche  cataloged 
B5006,  were  obtained  from  the  archives  of  the  Historical  Sec¬ 
tion,  Headquarters,  Air  Force  Logistics  Command,  Wright- 
Patterson  AFB,  OH. 


The  data  of  interest  included  total  fleet  flying  hours 
per  month  and  the  total  number  of  failures  by  WUC  per  month. 
The  B5006  catalog  contained  separate  reports  for  each  of  the 
two  models  of  the  F-16  (A  and  B)  covered  by  the  warranty. 

The  failure  data  from  each  report  was  thus  combined  for  this 
study.  The  catalog  contained  only  one  report  for  the  F-15, 
already  containing  a  compilation  of  data  for  both  the  F-15A 
and  F-15B.  The  failure  samples  for  this  study  consisted  of 
48  monthly  observations  for  each  aircraft  -  from  January  1975 
to  December  1978  for  the  F-15  and  from  January  1979  to  Decem¬ 
ber  1982  for  the  F-16.  This  latter  time  period  was  the 
actual  warranty  period  for  the  F-16  RIW.  The  time  interval 
selected  for  the  F-15  corresponded  to  the  similar  life  cycle 
phase  as  that  of  the  F-16  warranty  period.  The  selected  time 
periods  start  just  after  the  initiation  of  production  for 
each  aircraft.  The  F-15  became  operational,  with  the  deliv¬ 
ery  of  the  first  aircraft  to  the  Tactical  Air  Command  (TAC) 
in  November  1974;  the  first  F-16  was  delivered  to  TAC  in 
January  1979  (33:399,447). 

Shortcomings  of  AFM  66-1  Data .  The  Air  Force  MDCS  has 
often  been  maligned  as  an  inaccurate  and  incomplete  source  of 
maintenance  data.  In  fact,  in  1983,  "the  GAO  issued  a  report 
on  the  inadequacy,  inaccuracy,  and  inefficiency  of  the  USAF's 
D056/66-1  maintenance  data  collection  system"  (34:197).  One 
problem  with  the  system  is  that  it  does  not  allow  for  the 
direct  comparison  of  contractually  specified  MTBFs .  Only 
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MFHBFs  are  tracked,  not  MTBFs .  The  calculation  of  an  MTBF 
requires  that  total  operation  times,  both  ground  and  flight, 
be  known;  the  MDCS  only  tracks  flight  times.  The  system  does 
provide  an  indication  of  reliability  levels  and  trends,  how¬ 
ever.  Since  this  study  was  not  concerned  with  the  actual, 
absolute  reliability  levels  and  reliability  growths  of  each 
of  the  aircraft  but  with  the  relative  differences  between 
them,  this  problem  was  considered  irrelevant.  It  was  assumed 
for  this  study  that  the  ratio  of  ground  operating  time  to 
flight  time  was  the  same  for  both  aircraft. 

Another  problem  often  mentioned  about  the  MDCS  is  that 
maintenance  failure  data  is  reported  the  month  it  is  received 
and  not  the  month  the  failure  actually  occurred,  while  flight 
time  is  reported  the  month  it  is  recorded  (35:IIE-62).  For 
this  study,  it  was  assumed  that  the  lag  in  the  reporting  of 
failure  data  was  the  same  for  both  aircraft.  This  assumption 
plus  the  fact  that  this  analysis  covered  relatively  long 
periods  of  time  makes  this  second  problem  with  the  MDCS  also 
insignificant . 

A  third  often-cited  shortcoming  of  the  MDCS  is  inaccu¬ 
rate  data  (34:197;  35:IIE-54).  It  was  not  unreasonable  to 
assume  for  this  study  that  since  both  sets  of  failure  data 
came  from  the  same  data  collection  system,  the  F-15  data  was 
no  more  or  no  less  accurate  than  the  F-16  data. 

The  last  shortcoming  considered  for  this  study  was  the 
fact  that  the  MDCS  includes  many  failures  which  would  not  be 
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counted  as  failures  in  formal  reliability  demonstration  test¬ 
ing  (35:IIE-58;  36:466).  Examples  include  accidental  damage, 
tire  wearout,  and  minor  adjustments  (36:466).  Again,  there 
was  nothing  to  indicate  that  this  shortcoming  would  have  a 
different  impact  on  the  collection  of  F-15  versus  F-16  data. 
The  percentage  of  "insignificant  failures"  recorded  by  the 
MDCS  was  assumed  to  be  the  same  for  both  the  F-15  and  F-16. 

MDCS  Failure  Definition.  A  failure  was  defined  in 
general  terms  earlier  in  this  chapter.  The  MDCS  computer 
logic  for  determining  failures  is  not  that  simple,  however. 
The  MDCS  defines  failure  in  terms  of  How  Malfunctioned  (How 
Mai)  codes  and  action  taken  codes.  The  How  Mai  code  "con¬ 
sists  of  three  characters  and  is  used  to  identify  the  nature 
of  the  equipment  defect,  or  the  status  of  the  action  being 
accomplished"  (37:12).  These  codes  are  standard  throughout 
the  Air  Force.  The  action  taken  code  consists  of  one  charac¬ 
ter  and  is  used  to  identify  what  specific  work  was  done  on 
the  equipment  (37:11).  Failures  are  tracked  and  reported  by 
individual  WUCs. 

In  general,  the  MDCS  counts  an  item  as  a  failure  if 
maintenance  actions  take  place,  or  if  the  item  is  removed  and 
confirmed  defective.  Specifically,  the  MDCS  computer  defini¬ 
tion  of  a  failure  consists  of  the  following  two  decision 
rules  (32:5): 

1 .  Any  Type  1  How  Mai  code  in  combination  with  an 
action  taken  code  of  F,  K,  L,  or  Z. 
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2.  Any  Type  1  How  Mai  code  in  conjunction  with  an 
action  taken  code  of  P  or  R,  provided  the  removed  item  was 
not  found  serviceable  (action  taken  code  B)  at  the  bench 
check  station. 

A  Type  1  How  Mai  code  indicates  the  "item  no  longer 
meets  the  minimum  specified  performance  requirement  due  to 
its  own  internal  failure  pattern"  (38:18).  An  overwhelming 
majority  of  How  Mai  codes  are  designated  Type  1  (37:46).  The 

action  taken  codes  listed  above  are  defined  briefly  as 
follows  (37:11  ,12): 

F  -  Repair 

K  -  Calibrated  -  Adjustment  Required 

L  -  Adjust 

Z  -  Corrosion  Repair 

P  -  Removed 

R  -  Removed  and  Replaced 

B  -  Bench  Checked  Serviceable 

The  MDCS  computer  logic  definition  of  failure  was 
changed  in  March  1981  ( 35 : IIE-62 , 76 ) .  The  change  affected 

the  definition  of  failure  at  the  two  and  three  position  WUC 
level  ( 35 : IIE-76 , 77 , 79 ) .  But  since  all  LRUs  selected  for 
this  study  were  at  the  four  position  WUC  level  (i.e., 

Z  J.  h.  A  0 ) ,  it  was  assumed  that  this  logic  change  had  no  real 
impact  on  the  study. 


Research  Design  for  Research  Question  One 


Overview .  To  review,  the  first  research  question  was  - 
As  compared  with  the  reliabilities  of  nine  functionally  simi 
lar  non-warranted  LRUs  in  the  F-15,  are  the  reliabilities  of 
the  nine  F-16  warranted  LRUs  significantly  higher?  The 
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research  approach  used  to  answer  this  question  compared  the 
observed  reliabilities  (as  measured  by  MFHBF )  of  the  nine 

i 

warranted  F-16  LRUs  with  the  observed  reliabilities  (MFHBFs) 
of  functionally  similar  F-15  LRUs. 

The  following  is  a  brief  summary  of  the  steps  required 
to  answer  research  question  one: 

1.  Identify  the  non-warranted  F-15  LRUs  that  perform 

,  similar  functions  as  the  warranted  F-16  LRUS. 

I 

2.  Calculate  MFHBFs  for  the  selected  non-warranted  LRUs 
using  D056  data. 

3.  Calculate  MFHBFs  for  the  nine  warranted  LRUs  using 
I 

D056  data. 

4.  Perform  a  statistical  F  test  to  determine  whether 

the  MFHBFs  for  each  of  the  nine  F-16  warranted  LRUs  are  sig- 

I 

nificantly  higher  than  those  of  the  F-15  LRUs. 

For  the  basis  of  comparison,  the  third  year  of  produc- 
(  tion  MFHBFs  were  calculated  and  compared  for  each  aircraft. 

For  the  F-16,  this  was  January  to  December  1981,  and  for  the 
F-15,  January  to  December  1977.  The  third  year  was  selected 
for  comparison  because  it  was  assumed  for  this  study  that  any 
reliability  improvement  in  the  F-16,  as  a  direct  result  of 
the  RIW ,  would  have  already  been  incorporated  by  the  third 
year . 

Identif icaton  of  LRUs  for  Comparison.  Ten  LRUs  in  the 
F-15  with  similar  functions  as  the  nine  F-16  warranted  LRUs 
were  selected  for  this  study.  They  were  the  same  items  of 
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TABLE  I 


F-16  WARRANTED  LINE  REPLACEABLE  UNITS 


wuc 

F-16  LRU  Description 

Manufacturer 

1  4  A  AO 

Flight  Control  Computer 

Lear-Siegler ,  Inc. 

74AAO 

Radar  Antenna 

Westinghouse 

7  4  ABO 

Radar  Low  Power  Receiver 

Westinghouse 

74ACO 

Radar  Transmitter 

Westinghouse 

7  4  ADO 

Radar  Signal  Processor 

Westinghouse 

74AFO 

Radar  Computer 

Westinghouse 

7  4BAO 

Head-Up  (HUD)  Pilot  Display 

Marconi  Avionics 

74BCO 

HUD  Electronic  Unit 

Marconi  Avionics 

74DAO 

Inertial  Navigation  Unit  (INU) 

Singer-Kearfott 

Source:  (10:4-1;  39) 


equipment  included  in  the  ARINC  interim  evaluation  (29:2-7). 
Tables  I  and  II  list  the  warranted  F-16  and  the  selected  F-15 
LRUs  by  work  unit  code  and  LRU  description.  Note  that  the 
F-15  flight  control  computer  is  a  combination  of  two  separate 
LRUs.  The  descriptions  listed  came  from  the  work  unit  code 
manuals  for  each  aircraft.  The  F-15  and  F-16  have  only  one 
of  each  of  the  respective  19  LRUs  on  board. 

MFHBF  Calculations .  When  comparing  the  two  similar 
LRUs,  one  from  the  F-16  and  one  from  the  F-15,  one  must  ac¬ 
count  for  differences  in  design  specifications,  operational 
and  maintenance  environments,  and  other  related  factors  which 
may  affect  the  reliability  data.  It  was  reasonable  to  assume 
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TABLE  II 


EQUIVALENT  F-1 5  LINE  REPLACEABLE  UNITS 


F-1 5  WUC 

F-1 5  LRU  Description 

F-1 6  WUC 

52AAO  and  52ABO 

Flight  Control  Computer 

1  4AAO 

74FUO 

Radar  Antenna 

74AAO 

74FCO 

Radar  Receiver 

7  4  ABO 

74FAO 

Radar  Transmitter 

74ACO 

74FFO 

Radar  Target  Processor 

7  4  ADO 

74FQO 

Radar  Data  Processor 

74AFO 

7  4  KAO 

Head-Up  (HUD)  Pilot  Display 

74BAO 

74KCO 

HUD  Signal  Data  Processor 

74BCO 

71  AEO 

Inertial  Measurement  Unit  (IMU) 

7  4  DAO 

Source:  (40) 


for  this  study  that  the  operational  and  maintenance  environ¬ 
ments  of  the  two  aircraft  were  very  similar.  While  the  oper¬ 
ational  roles  of  the  aircraft  are  somewhat  different,  there 
was  no  reason  to  believe  this  difference  was  a  cause  for  con¬ 
cern  for  this  analysis.  The  possible  significant  factor  that 
must  be  addressed,  however,  is  the  design  difference  between 
each  of  the  comparable  LRUs  from  each  aircraft. 

The  LRUs  in  the  F-1 6  and  their  counterparts  in  the  F-1 5 
are  not  identical  pieces  of  equipment.  They  were  not  made  by 
the  same  manufacturer  and  they  did  not  contain  the  same  num¬ 
ber  of  SRUs  or  electronic  components.  The  comparable  LRUs 
did,  however,  perform  identical  functions.  This  latter  point 


was  important  to  the  development  of  this  methodology/  since 
it  has  been  shown  that  reliability  is  directly  related  to  a 
system's  function  (29:2-5).  "It  is  important,  therefore, 
that  only  LRUs  of  similar  function  be  compared  for  reliabil¬ 
ity  differences  that  might  arise  from  RIW  incentives" 

(29:2-5) . 

A  normalizing  technique  developed  in  the  ARINC  report 
was  used  to  account  for  the  differences  in  system  complexity 
between  the  comparable  F-15  and  F-16  LRUs.  The  technique  is 
based  on  the  fact  that  the  reliability  of  a  system  is  related 
to  the  number  of  components  that  make  up  the  system. 

Ideally,  the  total  number  of  all  the  electronic  parts  in  each 
of  the  LRUs  would  be  used  in  this  normalization  process. 
However,  since  that  kind  of  detailed  information  was  not 
available  to  the  researcher,  the  total  number  of  SRUs  in  each 
LRU  was  used  to  calculate  a  "complexity  factor"  (29:2-5). 

The  "complexity  factor"  is  defined  as  the  number  of  SRUs  in 
the  F-15  LRU  divided  by  the  number  of  SRUs  in  the  equivalent 
F-16  LRU  (29:2-5).  This  normalization  technique  is  based  on 
the  fact  that  the  failure  rate  for  an  entire  system  is  the 
sum  of  the  individual  failure  rates  of  the  components  that 
make  up  the  system  (41:76;  42:213).  The  following  assump¬ 
tions  also  apply  for  this  normalization  technique  to  be  valid 
(29:2-1 1  ) : 

1 .  The  electronic  equipment  failures  are  exponentially 
distributed,  and  the  same  "fundamental  failure  mechanisms 


exist  in  F-15  and  F-16  SRUs  and  in  the  same  proportion." 

2.  The  F-15  SRU  contains  the  same  average  number  of 
electronic  parts  as  the  F-16  SRU. 

3.  The  inherent  failure  rates  of  the  electronic  parts 
in  the  F-15  SRU  are  identical  to  the  failure  rates  of  the 
same  types  of  parts  in  the  F-16  SRU. 

The  F-15  and  F-16  Work  Unit  Code  Manuals  (T.O.  1F-15A-06 
and  T.O.  1F-16A-06)  list  all  the  SRUs  or  modules  that  make  up 
each  LRU.  The  only  manuals  available  for  this  study,  how¬ 
ever,  were  current.  Since  there  was  a  likely  chance  the  LRUs 
included  in  the  study  were  modified  sometime  between  now  and 
the  focal  time  periods  of  this  analysis,  the  current  WUC  man¬ 
uals  could  not  be  used  to  determine  the  number  of  SRUs  per 
LRU.  The  ARINC  interim  report  listed  the  number  of  SRUs  for 
each  of  the  LRUs  included  in  this  analysis;  however,  no  ref¬ 
erence  was  made  to  the  source  of  their  data.  Because  several 
substantial  differences  existed  between  the  numbers  reported 
in  the  ARINC  research  and  those  listed  in  the  current  WUC 
manuals,  the  following  method  was  devised  to  determine  how 
many  SRUs  made  up  each  of  the  19  LRUs.  The  D056  -  B5006 
report  lists  all  the  WUCs  on  file  in  the  master  record,  with 
SRUs  listed  under  the  parent  LRU  (32:1).  The  total  number  of 
different  SRUs  listed  for  each  of  the  ten  F-15  LRUs  during 
1976  and  1977  and  for  each  of  the  nine  F-16  LRUs  during  1980 
and  1981  was  used  to  calculate  the  "complexity  factor." 


Under  the  assumption  that  LRU  failures  are  exponentially 
distributed,  the  calculation  of  the  mean  life  or  MFHBF  is 
quite  straightforward.  The  maximum  likelihood  estimator,  9, 
of  the  mean  life  parameter  6  (MFHBF)  is  described  by 

9  =  T  /  r  (1  ) 

where  T  is  the  total  accummulated  flight  time  over  a  speci¬ 
fied  time  interval  and  r  is  the  total  number  of  failures 
observed  during  the  interval  (42:250,251).  The  MFHBFs  for 
both  F-15  and  F-16  LRUs  were  calculated  using  the  above  for¬ 
mula.  The  F-15  MFHBFs  for  each  LRU  were  then  normalized  by 
multiplying  this  figure  by  the  "complexity  factor"  (the  num¬ 
ber  of  F-15  SRUs  divided  by  the  number  of  F-16  SRUs) 
(29:2-11).  This  normalized  F-15  MFHBF  was  then  compared  with 
the  MFHBF  for  the  equivalent  F-16  LRU. 

Statistical  Comparison.  A  statistical  F  test  was  used 
to  compare  each  of  the  nine  F-16  LRU  MFHBFs  with  those  of  the 
equivalent  F-15  LRUs.  The  test  determined  whether  the  F-16 
MFHBFs  were  significantly  greater  than  those  of  the  F-15. 

The  hypothesis  in  this  case  was  directional,  and  a  one-tailed 
statistical  test  was  required  (43:353). 

The  alternate  hypothesis  (H  )  was  that  the  F-16  LRU 
MFHBF  is  statistically  greater  than  the  equivalent  normalized 


F-15  LRU  MFHBF: 


The  associated  null  hypothesis  (Hq)  was  that  there  is  no  dis¬ 
cernible  difference  between  the  F-16  and  F-15  LRU  MFHBFs : 

Ho  :  *16  =  9  15 

The  decision  rule  for  conducting  the  test  was  based  on  a  five 
percent  (.05)  level  of  significance. 

The  development  of  the  F  test  statistic  was  based  on  the 

fact  that  for  the  exponential  distribution  the  quantity  2r e/9 

2 

is  chi-square  ( X  )  distributed  with  2r  degrees  of  freedom 
(42:267).  It  was  also  based  on  the  fact  that  the  ratio  of 
two  independent  chi-square  random  variables  (X1  and  X2 )  de¬ 
fines  the  following  F  distribution  (44:70): 

F  =  \  Xi  1  7i  X2  (2) 

where  Xi  has  7  degrees  of  freedom  and  X2  has  72  degrees  of 
freedom.  By  substituting  2r ,  for  X^  ,  2r 2  ®2  /  02  for  X2  f 

2ri  for  7  ,  and  2r2  for  T2  in  Eq  (2)  the  following  results 
after  simplification: 

F  =  0,  /  02  (3) 

Since  the  null  hypothesis  equates  6^  and  9  ,  Eq  (3)  reduces 
to 

F  =  8 1  /  9  2  (  4  ) 

which  is  an  F  distribution  with  2r  ,  2r  degrees  of  freedom. 


The  test  statistic  in  this  specific  case  was  thus 


F  =  §  =  F-1 6  MFHBF  /  F-15  Normalized  MFHBF  (5) 

C  1 6  •  5 

The  rejection  decision  criteria  was  the  following:  if 
the  calculated  value  of  F  (F  )  is  greater  than  the  critical 

C 

value  of  F,  reject  the  null  hypothesis  in  favor  of  the  alter¬ 
nate  hypothesis  at  a  .05  significance  level  (43:353).  The 
critical  value  of  F  (from  tables  of  the  percentage  points  of 
the  F  distribution)  was  based  on  2(r,  -1),  2(r,  -1)  degrees 
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of  freedom  rather  than  2r1g  and  2ri5  because  one  parameter 
(9)  had  been  calculated  from  the  r  failure  variables  (19:52). 
The  rejection  of  the  null  hypothesis  would  answer  the  first 
research  question  in  the  affirmative. 

A  corollary  hypothesis  test  was  used  to  determine 
whether  the  F-15  MFHBF  was  significantly  greater  than  the 
MFHBF  of  the  F-1 6  in  those  cases  where  the  null  hypothesis  in 
the  above  test  could  not  be  rejected.  The  corollary  alter¬ 
nate  hypothesis  was  thus 


The  null  hypothesis  was  identical  to  that  above;  that  there 
is  no  difference  between  the  F-1 6  and  F-15  LRU  MFHBFs: 


H  : 
0 


e 


15 


The  decision  rule  for  conducting  this  corollary  test  was  also 
based  on  a  five  percent  (.05)  level  of  significance.  The 


test  statistic  was 


Again,  the  rejection 


decision  criteria  was  the  following:  if  F  is  greater  than 

c 

the  critical  value  of  F,  from  the  F  tables,  then  reject  the 

null  hypothesis  in  favor  of  the  alternate  hypothesis  at  a  .05 

significance  level.  The  critical  value  of  F  was  based  on 

2(r  -1),  2(r  -1)  degrees  of  freedom. 

15  16 

Confidence  Intervals .  Confidence  intervals  provide  a 
measure  of  the  uncertainty  regarding  the  MFHBF  estimates. 
Kapur  and  Lamberson  have  shown  that  for  reliability  studies 
where  only  the  number  of  failures  over  a  time  interval  have 
been  recorded,  the  following  100(1— o;)%  two-sided  confidence 
interval  applies  (42:254): 


2T  /  X3 

a 


/2»  2(r+1) 


<  9  < 


2T  /  X 


a 

1-o/2,  2r 


(6) 


where  T  equals  the  interval  of  time  and  r  equals  the  number 
of  observed  failures  over  that  time  interval  and  the  two 
denominators  represent  the  chi-square  table  values  with  the 
appropriate  significance  levels  and  degrees  of  freedom.  For 
this  study  90%  confidence  intervals  for  the  F-15  and  F-16 
LRUs  were  calculated  using  Eq  (6).  Normalized  intervals  were 
calculated  for  the  F-15  by  multiplying  T  by  the  "complexity 
factor"  discussed  above,  before  using  Eq  (6). 


Research  Design  for  Research  Question  Two 

Overview.  To  review,  the  second  research  question  was  - 
As  compared  with  functionally  similar  non-warranted  LRUs  in 
both  the  F-15  and  F-16,  did  the  nine  F-16  warranted  LRUs 


r.  i 


37 


exhibit  a  significantly  higher  reliability  growth  during  the 
first  four  years  of  operation?  The  research  approach  used  to 
answer  the  second  question  compared  the  reliability  (MFHBF) 
growths  of  the  nine  warranted  F-16  LRUs  with  the  MFHBF 
growths  of  the  ten  non-warranted  F-15  LRUs  and  also  with  the 
combined  MFHBF  growth  of  nine  dissimilar  non-warranted  F-16 
LRUs,  over  an  equivalent  four  year  period. 

The  following  briefly  describes  the  steps  developed  to 
answer  research  question  two: 

1.  Identify  nine  non-warranted  electronic  F-16  LRUs 
that  are  similar  in  complexity  to  the  warranted  LRUs. 

2.  Calculate  the  reliability  growth  parameters  for  each 
of  the  F-15  and  the  warranted  F-16  LRUs,  using  the  Army 
Materiel  Systems  Analysis  Activity  ( AMS A A )  Reliability  Growth 
Model  as  a  basis. 

3.  Calculate  the  reliability  growth  parameters  for  both 
a  combined  total  of  the  warranted  F-16  LRUs  and  a  combined 
total  of  the  non-warranted  F-16  LRUs. 

4.  Perform  a  statistical  F  test  using  the  growth  param¬ 
eters  to  determine  whether  the  reliability  growths  exhibited 
by  the  warranted  F-16  LRUs  are  significantly  higher  than 
those  exhibited  by  non-warranted  LRUs  in  both  the  F-15  and 
F-16. 

Equivalent  four  year  time  intervals  were  selected  for 
both  aircraft  as  a  basis  for  the  reliability  growth  compar¬ 
ison.  For  the  F-16,  the  interval  corresponded  to  the  actual 
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warranty  period,  January  1979  to  December  1982.  For  the 
F-15,  the  equivalent  period  was  January  1975  to  December 
1978.  Research  has  shown  that  reliability  improvements  which 
are  a  direct  result  of  a  RIW  rarely  occur  after  the  two  year 
point  of  the  warranty  (17:98).  Despite  this,  the  entire  four 
year  warranty  period  and  the  equivalent  F-15  time  period  were 
selected  because  the  effects  of  the  improvements  in  terms  of 
MFHBF  growth  are  often  not  observed  "until  the  percentage  of 
improved  equipment  becomes  a  substantial  part  of  the  entire 
population";  which  may  take  several  years  (36:466). 

In  addition  to  the  F-16  and  F-15  LRUs  identified  for 
research  question  one,  nine  other  non-warranted  F-16  LRUs 
were  selected  for  the  second  part  of  this  analysis.  For  the 
most  part,  the  LRUs  selected  were  similar  in  complexity  to 
the  warranted  LRUs,  and  most  were  items  of  electronic  avion¬ 
ics  equipment.  Table  III  lists  these  non-warranted  F-16 
LRUs.  The  first  three  LRUs  listed  were  the  three  that  were 
dropped  from  the  original  12  LRUs  considered  for  the  F-16  RIW 
(45:223).  The  LRU  descriptions  are  those  listed  in  the  F-16 
WUC  Manual.  Again,  these  nine  LRUs  were  also  included  in  the 
ARINC  interim  evaluation  (29:2-7). 

Reliability  Growth  Modeling.  The  choice  of  how  to  ana¬ 
lyze  reliability  growth  data  is  probably  best  made  after  the 
data  has  been  collected  and  studied.  In  reality,  the  data 
analysis,  or  more  specifically,  the  reliability  growth  model¬ 
ing,  may  be  more  an  art  than  a  science.  For, 


"no  model  can 


TABLE  III 


F-1 6  NON-WARRANTED  LRUs  INCLUDED  IN  THE  STUDY 

_ WUC _ LRU  Description _ 

74CAO  Fire  Control  Computer 

74EAO  Radar  Electro-Optical  Display 

74EB0  Radar  Electro-Optical  Electronics 

1 4ADO  Flight  Control  Panel 

1 4FB0  Air  Data  Electronic  Components 

74DDO  Fire  Control  Nav.  Panel 

75DAO  Stores  Control  Panel 

75DBO  Missile  Remote  Interface  Unit 

75DCO  Stores  Mgt.  Central  Interface  Unit 

Source:  (39) 

completely  characterize  the  reliability  evolution  of  a  comp¬ 
lex  system  design"  (46:43).  The  underlying  basis  for  relia¬ 
bility  growth  modeling  is  the  fact  that  successive  times 
between  failures  will  tend  to  increase  in  the  presence  of 
improving  reliability  (47:113).  Numerous  models  have  been 
developed  to  represent  the  reliability  growth  of  a  wide  va¬ 
riety  of  systems.  In  fact,  Military  Handbook  189  ( MIL-HDBK- 
189),  entitled  Reliability  Growth  Management ,  describes  17 
different  models  (47).  Of  these  17,  the  two  most  commonly 
used  are  the  Duane  Model  and  the  Army  Materiel  Systems  Analy 
sis  Activity  (AMSAA)  Model  (22:122).  MIL-HDBK-189  suggests 
the  Duane  Model  is  best  used  during  planning  phases,  while 


the  AMS A A  Model  is  best  used  for  assessment  and  tracking 
(22:122).  The  Duane  Model's  advantage  stems  from  its  math¬ 
ematical  simplicity;  while  the  advantages  of  the  AMSAA  Model 
are  its  versatility  and  its  associated  statistical  properties 
(22:122).  The  AMSAA  Model  "has  found  wide  acceptance,  espe¬ 
cially  among  the  military  and  military  contractors"  (48:20), 
and  is  the  model  recommended  by  MIL-HDBK-189  because  it  is 
"the  most  versatile  for  tracking  growth"  (22:43).  Because  of 
this  and  the  fact  that  the  ARINC  interim  study  demonstrated 
the  usefulness  of  the  model  in  assessing  the  reliability 
growths  of  F-15  and  F-16  LRUs,  the  AMSAA  Model  was  used  to 
evaluate  the  LRU  reliability  growths  in  this  analysis. 

AMSAA  Reliability  Growth  Model .  Before  discussing  the 
AMSAA  Model,  a  brief  explanation  of  the  Duane  Model  is  re¬ 
quired,  since  the  Duane  postulate  really  led  to  the  develop¬ 
ment  of  the  AMSAA  Model  (22:54). 

In  1962,  J.  T.  Duane,  of  the  General  Electric  Company, 
developed  probably  the  most  popular  model  of  reliability 
growth  (48:11;  49:390).  He  postulated  that  an  approximate 
linear  relationship  existed  between  the  logarithm  of  the 
cumulative  failure  rate  (or  cumulative  MTBF)  and  the  loga¬ 
rithm  of  the  cumulative  operating  time  (22:47).  In  other 
words,  the  plot  of  cumulative  failure  rate  against  cumulative 
time  would  approximate  a  straight  line  on  log-log  paper.  The 
cumulative  failure  rate  is  defined  as  the  cumulative  number 
of  failures  divided  by  the  cumulative  test  time,  and  is  the 


inverse  of  the  cumulative  MTBF  for  system  failures  following 
an  exponential  distribution.  Equation  (7)  expresses  the 
above  relationship  mathematically: 


In  \  (t)  =  In  K  -  din  t  (7) 

C 

Through  simplification,  Duane's  Growth  Model  is  represented 
in  the  following  form  (46:9,10;  48:12,13): 

Xc(t)  «  Kt*  (8) 

where 

A  (t)  =  the  cumulative  failure  rate 

c 

t  =  the  cumulative  test  time 

K  =  the  cumulative  failure  rate  at  t  =  0,  a  constant 

or  =  the  Duane  growth  rate  constant,  the  negative  of 
the  slope  of  the  growth  curve 

Although  the  two  parameters,  K  and  a,  can  be  estimated  using 
standard  curve  fitting  methods,  such  as  the  least  squares 
method,  the  Duane  Model  is  probably  more  useful  when  applied 
as  a  graphical  technique  (46:10;  48:13).  When  plotted  on 
log-log  paper,  K  is  estimated  by  the  intercept  of  the  line 
and  a  is  estimated  by  the  line's  slope  (48:14).  The  AMSAA 
Model,  while  somewhat  more  complicated  than  the  Duane  Model, 
enables  one  to  go  a  step  further  by  calculating  maximum  like¬ 
lihood  estimators  of  its  parameters. 

The  AMSAA  Model  was  developed  by  Larry  H.  Crow,  a  re¬ 
searcher  working  for  the  U.S.  Army  Materiel  Systems  Analysis 
Agency.  Crow  used  the  Duane  postulate  (the  plot  of  the 
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logarithm  of  the  cumulative  failure  rate  versus  the  logarithm 
of  the  cumulative  time  is  linear)  in  the  empirical  develop¬ 
ment  of  his  model  (22:54).  Crow  has  shown  that  the  Dua_. 
growth  model  is  mathematically  equivalent  to  a  Weibull  proc¬ 
ess  where  failure  times  are  assumed  to  occur  according  to  a 
nonhomogeneous  Poisson  process  with  the  following  Weibull 
intensity  function  (50:438,439): 

r(t)  =  A^t*3-1  ,  where  A ,  0  >  0  (9) 

In  K 

where  A,  the  scale  parameter,  equals  e  (where  In  K  is  from 
Eq  (7)),  and  /3  is  the  shape  or  growth  parameter  and  is  re¬ 
lated  to  the  slope  of  the  growth  curve  (22:55).  As  such,  0 
reflects  the  rate  at  which  reliability  ( MFHBF )  increases  or 
decreases  (50:439).  The  Weibull  process  growth  parameter,  /3, 
is  related  to  the  Duane  Model  by  the  following:  0  =  1  +  a  , 
where  a  is  the  Duane  growth  rate  constant  from  Eq  (8) 

(22:57).  Because  a  is  the  negative  of  the  slope,  the  slope 
of  the  cumulative  failure  rate  versus  cumulative  time  curve 
plotted  on  log-log  paper  is  given  by  1  -  0 .  When  0  is  less 
than  one,  the  failure  rate,  r(t),  is  decreasing,  indicating 
reliability  growth;  when  0  is  greater  than  one,  r(t)  is 
increasing,  implying  some  form  of  system  reliability  deterio¬ 
ration  (47:127;  51:385). 

The  AMSAA  Model  uses  a  nonhomogeneous  Poisson  process 
(NHPP),  which  is  often  termed  a  "Weibull  Process,"  to  model 
reliability  growth  (52:426).  It  is  important  to  note  that 


this  does  not  imply  the  use  of  the  Weibull  distribution. 


"Therefore,  statistical  procedures  for  the  Weibull  distri¬ 
bution  do  not  apply"  to  the  AMSAA  Model  (47:127).  Very 
briefly,  the  difference  between  an  NHPP  and  a  homogeneous 
Poisson  process  (HPP)  is  that  the  failure  rate  varies  with 
time  for  the  NHPP  and  is  constant  for  a  HPP  (52:428). 

Growth  Parameter  Calculations .  The  use  of  the  AMSAA 
Model  permits  the  estimation  of  the  model  parameters  by  sta¬ 
tistical  means.  The  method  of  maximum  likelihood  provides 
estimates  of  both  the  shape  parameter,  0,  and  the  scale 
parameter,  \  (47:139).  Since  individual  failure  times  were 
not  available  through  the  MDCS  D056  system,  and  only  the 
flying  times  and  the  number  of  failures  per  month  were  ob¬ 
tainable,  the  procedures  outlined  in  MIL-HDBK-189  for  grouped 
data  were  used  for  this  analysis.  Using  these  procedures, 
the  estimation  of  0  and  A  is  possible  through  maximum  likeli¬ 
hood  methods. 

Parameter  estimation  from  grouped  data  is  based  on  the 
assumption  that  there  are  k  intervals  of  time  which  are 
defined  by  the  boundaries  tQ,  t1 ,  t2,....,  t^ ,  where  tQ  *  0 
(29:A-3).  The  time  intervals  do  not  necessarily  have  to  be 
equal  in  length  (47:139).  The  number  of  failures  observed  in 
the  interval  from  ti-t  to  t^.  is  denoted  by  Ni.  The  following 
equation  from  MIL-HDBK-189  was  used  to  calculate  0,  the  maxi¬ 
mum  likelihood  estimate  of  0: 
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where  t  In  t  is  defined  as  zero  (47:139).  This  non-lin- 

o  o 

ear  equation  can  be  solved  by  any  number  of  numerical  analy¬ 
sis  iterative  procedures.  For  this  study,  the  secant  method 
was  used.  The  general  iteration  formula  for  this  method  is 


n+1 


=  x 


-  f(x  ) 
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f (X  )  - 
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(11 ) 


where  n  >  1  (53:80).  For  this  study,  the  researcher  devel¬ 
oped  a  Hewlett-Packard  41 C  calculator  program,  using  Eqs  (10) 
and  (11),  to  calculate  0  to  four  decimal  places.  The  growth 
parameter  of  the  F-15  LRU  was  then  compared  with  that  of  the 
equivalent  F-16  LRU  to  determine  whether  there  was  a  statis¬ 
tical  difference  between  the  two. 

Statistical  Comparison.  Again,  a  statistical  F  test  was 
used  to  compare  the  growth  parameters,  0,  of  the  F-15  and  the 
F-16  LRUs  and  the  warranted  and  non-warranted  F-16  LRUs. 

Like  the  hypothesis  in  research  question  one,  this  hypothesis 
was  also  directional,  so  a  one-tailed  statistical  test  was 
required. 

Since  the  slope  of  the  cumulative  failure  rate  versus 
cumulative  time  curve  plotted  on  log-log  paper  is  equal  to 
1  -  H,  the  smaller  0  is,  the  greater  the  slope  is.  And  the 
greater  the  slope  of  the  line  is,  in  an  absolute  sense,  the 


greater  is  the  reliability  growth  experienced  by  the  system. 
Thus,  the  LRU  experiencing  the  greater  reliability  growth  is 
the  one  with  the  smaller  growth  parameter,  13.  Since  only  the 
differences  in  the  growth  parameters  were  of  interest  to  an¬ 
swer  research  question  two,  there  was  no  need  to  normalize 
the  F-15  LRUs  to  a  common  complexity  level. 

F-1 5  Versus  F-1 6 .  The  alternate  hypothesis  for  the 
first  part  of  this  comparison  was  that  the  warranted  F-1 6  LRU 
reliability  growth  was  statistically  greater  than  the  equiva¬ 
lent  F-15  LRU  reliability  growth. 

V  *16  <  *15 

The  associated  null  hypothesis  was  that  there  was  no  discern¬ 
ible  difference  between  the  reliability  growths  of  the  war¬ 
ranted  F-1 6  and  non-warranted  F-15  LRUs. 


The  test  was  based  on  a  five  percent  (.05)  level  of  signifi¬ 
cance.  To  test  whether  the  growth  parameters  were  the  same, 
the  following  test  statistic,  developed  by  Crow,  was  used 
(51 : 398 ) : 

Fc'»,5/5,6  <12» 

which  is  based  on  2N.,_  ,  2N  _  degrees  of  freedom,  where  N  is 
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the  total  number  of  failures  over  four  years  for  the  respec¬ 
tive  LRU  (29:A-6;  51:398).  The  rejection  decision  criteria 


was  the  following:  if  the  calculated  value  of  F  (Fc)  is 
greater  than  the  critical  value  of  F,  reject  the  null  hypoth¬ 
esis  in  favor  of  the  alternate  hypothesis  at  a  .05  signifi¬ 
cance  level  (43:353).  The  critical  value  of  F,  from  the  F 
tables,  was  based  on  2N  ,  2N1g  degrees  of  freedom.  The 
rejection  of  the  null  hypothesis  would,  in  part,  answer  the 
second  research  question  in  the  affirmative. 

As  in  the  methodology  for  the  first  research  question, 
in  those  cases  where  the  null  hypothesis  in  the  above  test 
could  not  be  rejected,  a  corollary  hypothesis  test  was  used 
to  determine  whether  the  F-15  reliability  growth  was  signifi¬ 
cantly  greater  than  that  of  the  F-16.  The  corollary  alter¬ 
nate  hypothesis  was  thus 

Ha:  015  <  0 16 

The  null  hypothesis  was  identical  to  that  above;  that  there 
is  no  difference  between  the  F-16  and  F-15  LRU  MFHBF  growth. 

V  0  15  *  0  16 

The  decision  rule  for  conducting  this  corollary  test  was  also 
based  on  a  five  percent  level  of  significance.  The  test  sta¬ 
tistic  was  F^  =  (3  /  0  .  Again,  the  rejection  decision 

criteria  was  the  following:  if  Fc  is  greater  than  the  criti¬ 
cal  value  of  F,  then  reject  the  null  hypothesis  in  favor  of 
the  alternate  hypothesis  at  a  .05  significance  level.  The 


critical  value  of  F  was  based  on  2N,_  ,  2N,_  degrees  of 

16  15 

freedom. 

F-1 6  RIW  Versus  F-1 6  Non-RIW .  The  alternate  hy¬ 
pothesis  for  the  second  part  of  the  comparison  was  that  the 
reliability  growth  of  the  combination  of  the  entire  group  of 
warranted  F-1 6  LRUs  (the  RIW  "system")  was  statistically 
greater  than  that  of  the  combination  of  the  entire  group  of 
non-warranted  F-1 6  LRUs  (the  Non-RIW  "system"): 

H  :  B  <8 

a*  H  RIlil  M  IMONRIUJ 

The  associated  null  hypothesis  for  this  second  part  was  that 
there  was  no  discernible  difference  between  the  reliability 
growth  of  the  RIW  "system"  and  that  of  the  Non-RIW  "system": 

JJ  •  Q  =  /3 

o*  RlbJ  p  NONRIlii 

The  decision  rule  for  conducting  the  hypothesis  tests  was 
based  on  a  five  percent  (.05)  level  of  significance. 

For  this  second  comparison,  the  failures  of  all  the  RIW 
LRUs  were  totaled  for  each  month,  and  the  growth  parameter  of 
this  group  as  a  whole  was  calculated  using  Eq  (10).  The  same 
applied  for  the  group  of  non-warranted  F-1 6  LRUs.  The  idea 
was  to  neutralize  "any  technological  differences  that  might 
be  thought  to  exist  between  F-1 6  and  F-1 5  aircraft" 

(29:2-18) . 

The  F  statistic  used  to  test  whether  the  growth  param¬ 
eters  were  the  same  was  Fc  =  $  mmlJ  0RIU)  <51  :398), 


which  is  based  on  2nW]NRI1i,/  2Nriu  degrees  of  freedom,  where  N  is 
the  total  observed  number  of  failures  for  the  respective 
group  (29:A-6;  51:398).  Again,  the  rejection  decision  cri¬ 
teria  was  the  following:  if  the  calculated  value  of  F  (F  ) 

C 

is  greater  than  the  critical  value  of  F,  reject  the  null  hy¬ 
pothesis  in  favor  of  the  alternate  hypothesis  at  a  .05  sig¬ 
nificance  level.  The  critical  value  of  F  (from  tables  of  the 
percentage  points  of  the  F  distribution)  was  based  on 
2Nriui  degrees  of  freedom.  The  rejection  of  the  null  hypothe¬ 
sis,  in  conjunction  with  the  rejection  of  the  null  hypothesis 
of  part  one  of  this  comparison,  would  answer  the  second  re¬ 
search  question  in  the  affirmative. 

Research  Assumptions  and  Limitations 

For  the  purposes  of  this  analysis,  the  researcher  made  . 
several  assumptions  and  also  identified  certain  limitations 
which  had  impacts  on  the  analysis.  The  assumptions  were 
categorized  as  being  either  general  or  statistical  in  nature. 
The  assumptions  and  limitations  were  necessary  for  research 
consistency,  and  are  explained  below. 

Assumptions.  The  following  list  reflects  the  general 
assumptions  applicable  to  this  methodology: 

1 .  Mean  Flight  Hours  Between  Failures  data  was  a  valid 
measure  of  system  reliability. 

2.  The  AFM  66-1  Maintenance  Data  Collection  System 


accurately  measured  the  actual  failures  and  flying  time  for 
the  selected  LRUs. 


3.  The  failure  of  any  one  of  the  LRUs  had  no  effect  on 
the  operation  of  the  other  LRUs  in  the  study. 

4.  The  differences  in  the  operational  environments  and 
the  maintenance  levels  of  the  F-15  and  F-16  were  insignifi¬ 
cant. 

The  statistical  assumptions  applicable  to  this  analysis 
are  the  following: 

1 .  The  reliability  of  the  equipment  included  in  the 
study  followed  the  exponential  law;  failures  were  chance 
failures  and  were  distributed  exponentially. 

2.  LRU  failures  were  independent  of  each  other. 

3.  The  use  of  the  system  "complexity  factor"  was  a 
valid  means  of  normalizing  the  F-15  data. 

4.  The  AMS A A  Reliability  Growth  Model  adequately  repre¬ 
sented  the  reliability  growth  of  the  LRUs  included  in  the 
study . 

Limitations.  To  adequately  research  all  the  areas  re¬ 
lated  to  the  effectiveness  of  the  F-16  reliability  improve¬ 
ment  warranty  was  clearly  impossible  given  the  time  con¬ 
straints.  The  research  design  only  allowed  the  researcher  to 
make  inferences  about  the  reliability  and  reliability  growth 
of  the  F-16  LRUs  included  in  the  RIW.  Other  measures  of 
effectiveness,  especially  cost  factors,  were  not  addressed. 

No  attempt  was  made  to  analyze  the  reliability  perform¬ 
ance  of  the  European  Participating  Governments'  aircraft 
included  in  the  F-16  RIW. 


Analysis  of  Results 


This  chapter  details  the  results  of  the  statistical 
analyses  of  the  two  research  questions  using  the  methodology 
described  in  Chapter  II.  The  appropriate  F  test  statistics 
were  calculated  using  the  data  collected  from  the  Air  Force 
MDCS;  then  these  values  were  compared  to  the  critical  F  val¬ 
ues  to  determine  the  acceptance  or  rejection  of  the  null  hy¬ 
potheses.  To  provide  consistency  in  the  review  of  the  sta¬ 
tistical  results,  each  of  the  hypotheses  for  each  research 
question  is  discussed  individually. 

As  discussed  in  the  previous  chapter,  the  source  of  the 
data  for  this  analysis  was  the  D056  system  of  the  AFM  66-1 
MDCS.  The  B5006  catalog  contained  the  specific  F-15  and  F-16 
reports  required  for  the  study.  The  F-15  and  F-16  reports 
were  filed  according  to  the  air  logistics  center  (ALC)  re¬ 
sponsible  for  the  aircraft.  In  this  case,  the  F-15  reports 
were  filed  under  the  Warner  Robins  ALC  and  those  for  the  F-16 
were  filed  under  the  Ogden  ALC.  Failure  and  flight  time  data 
were  extracted  from  these  reports  and  are  summarized  in 
Appendix  A. 


First  Research  Question 


To  review,  the  first  research  question  was  -  As  compared 


with  the  reliabilities  of  nine  functionally  similar  non-war- 
ranted  LRUs  in  the  F-15,  are  the  reliabilities  of  the  nine 


F-16  warranted  LRUs  significantly  higher? 

To  answer  research  question  one,  the  observed  MFHBF  of 
each  of  the  nine  warranted  F-16  LRUs  was  compared  to  that  of 
its  equivalent  F-15  LRU(s).  The  MFHBF s  for  the  F-16  LRUs 
were  calculated,  using  the  total  failure  and  total  flight 
time  data  for  1981.  The  F-15  LRU  MFHBFs  were  calculated 
using  the  data  for  1977.  Normalized  F-15  MFHBFs  were  then 
calculated  by  multiplying  the  raw  F-15  MFHBFs  by  the  appro¬ 
priate  "complexity  factor."  These  "complexity  factors",  the 
ratios  of  the  number  of  F-1 5  SRUs  per  LRU  to  the  number  of 
F-16  SRUs  per  equivalent  LRU,  are  listed  in  Table  IV.  This 
number  of  SRUs  per  LRU  or  SRU  count  was  derived  from  the 
D056  -  B5006  reports.  Table  V  lists  the  results  of  the  MFHBF 
calculations  for  the  nine  LRU  comparisons.  Both  raw  and 
normalized  F-15  MFHBFs  are  listed. 

A  statistical  F  test  was  used  to  determine  whether  the 
F-16  LRU  MFHBFs  were  significantly  greater  than  those  of  the 
equivalent  F-15  LRU's,  at  the  five  percent  significance  or 
alpha  level.  The  null  hypothesis  stated  that  there  was  no 
discernible  difference  and  the  alternate  hypothesis  stated 
that  the  F-16  LRU  MFHBF  was  greater  than  the  equivalent  F-15 


LRU  MFHBF: 


TABLE  IV 


NUMBER  OF 

SRUs  PER  LRU 

LRU  Description 

F-1  6  WUC 

F-1  6 
SRU 
Count 

F-1 5  WUC 

F-1  5 
SRU 
Count 

C.F. 

Flight  Control 

1  4AA0 

1 1 

52AAO 

22 

2.00 

Computer 

5  2  ABO 

Radar  Antenna 

74AAO 

6 

7  4FUO 

1  0 

1  .67 

Radar  Receiver 

7  4  ABO 

1 1 

74FCO 

6 

.55 

Radar  Transmitter 

74ACO 

8 

7  4FAO 

1  4 

1  .75 

Radar  Signal  Proc. 

74ADO 

23 

74FFO 

25 

1  .09 

Radar  Computer 

74AFO 

13 

74FQO 

30 

2.31 

HUD  Pilot  Display 

74BAO 

9 

7  4  KAO 

17 

1  .89 

HUD  Electronics 

74BCO 

13 

7  4KCO 

17 

1  .31 

Inertial  Nav.  Unit 

74DAO 

1  4 

71AEO 

4 

.29 

C.F.  =  "Complexity 

Factor"  = 

F-1 5  SRU  Count  / 

F-1 6  SRU 

Count 

Source: 

(54) 

The  summary  of  the 

F  test  results  is 

given  in 

Table  VI. 

■  Noti 

that  since  percentage  points  of  the  F  distribution  tables 
only  list  values  for  degrees  of  freedom  up  to  120,  an  approx 
imation  formula  was  used  to  calculate  the  critical  F  value. 
Appendix  B  summarizes  the  technique  used. 

The  results  from  Table  VI  indicated  that  the  null  hy¬ 
pothesis  of  equal  MFHBFs  can  be  rejected  for  six  of  the  nine 
LRU  comparisons.  Since  the  test  will  lead  to  an  incorrect 
conclusion  only  five  percent  of  the  time  (43:288),  it  was 


LRU  Description 

Fc 

F  .05 

Decision 

Flight  Control  Comp. 

.52 

1  .21 

test  further 

Radar  Antenna 

.60 

1  .13 

test  further 

Fadar  Receiver 

2.02 

1.14 

rej  ect 

Radar  Transmitter 

1  .49 

1.15 

reject 

Radar  Signal  Proc. 

3.13 

1  .17 

reject 

Radar  Computer 

1.71 

1.17 

reject 

HUD  Pilot  Display 

1  .06 

1  .17 

do  not  reject 

HUD  Electronics 

2.39 

1  .25 

rej  ect 

Inertial  Nav.  Unit 

4.72 

1.12 

reject 

concluded  that  the  radar 

receiver , 

transmitter 

,  signal  proc- 

essor,  and  computer  along 

with  head 

-up  display 

electronic 

unit  and  the  inertial  navigation  unit,  all  exhibited  signifi¬ 
cantly  greater  MFHBFs  than  their  counterparts  in  the  F-15. 
Since  the  null  hypothesis  could  not  be  rejected  for  the  HUD 
pilot  display,  it  was  concluded  there  was  no  significant 
difference  between  the  MFHBFs  of  these  two  LRUs. 

As  discussed  in  Chapter  II,  a  corollary  hypothesis  test 
was  planned  to  compare  the  MFHBFs  of  those  pairs  of  LRUs  that 
could  not  be  rejected  under  the  initial  hypothesis  test. 

This  corollary  null  hypothesis  stated  that  there  was  no  dis¬ 
cernible  difference  between  the  MFHBFs  and  the  alternate 
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TABLE  VII 


SUMMARY  OF  F  TEST  RESULTS  -  COROLLARY  MFHBF  COMPARISON 


LRU  Description 

Fc 

F  ,05 

Decision 

Flight  Control  Comp. 

1  .93 

1  .21 

rej  ect 

Radar  Antenna 

1  .67 

1.13 

rej  ect 

hypothesis  stated  that  the  F-16  LRU  MFHBF  was  less  than  the 
equivalent  F-15  LRU  MFHBF: 


H  : 

0 

=  6  _ 

0 

16 

15 

H  : 

0 

<  0 

a 

16 

15 

The  summary  of  the  F  test  results  for  the  corollary  hypoth¬ 
esis  is  given  in  Table  VII.  The  results  indicated  that  the 
null  hypothesis  of  equal  MFHBF s  can  be  rejected  for  both  of 
the  LRU  comparisons.  It  was  thus  concluded  that  the  F-15 
flight  control  computer  and  radar  antenna  exhibited  signifi¬ 
cantly  greater  MFHBFs  than  their  warranted  counterparts  in 
the  F-16. 

Ninety  percent  confidence  intervals  were  calculated, 
using  the  equation  given  in  Chapter  II,  for  the  warranted 
F-16  LRU  MFHBFs  as  well  as  the  equivalent  normalized  F-15  LRU 
MFHBFs.  An  inspection  of  these  confidence  intervals,  listed 
in  Table  VIII,  yields  the  same  conclusions  as  the  statistical 


F  tests  discussed  above. 
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TABLE  VIII 


F-16  AND  NORMALIZED  F-15 

MFHBF  90% 

CONFIDENCE 

INTERVALS  (Cl) 

LRU  Description 

F-1  6 

MFHBF  Cl 

F-15 

NORMALIZED 
MFHBF  Cl 

Flight  Control  Comp. 

(310, 

399) 

(583, 

789) 

Radar  Antenna 

(127, 

1  49) 

(209, 

253) 

Radar  Receiver 

(150, 

179) 

(  73, 

89) 

Radar  Transmitter 

(251  , 

315) 

(174, 

206) 

Radar  Signal  Proc. 

(343, 

448) 

(115, 

1  36) 

Radar  Computer 

(345  , 

451  ) 

(213, 

250) 

HUD  Pilot  Display 

(300, 

385) 

(289, 

357) 

HUD  Electronics 

(637, 

932) 

(283, 

366) 

Inertial  Nav.  Unit 

(124, 

144) 

(  26, 

31  ) 

In  summary,  then,  of  the  nine  warranted  F-16  LRUs,  six 
exhibited  significantly  greater  third  year  MFHBFs  and  thus, 
better  reliability,  than  their  F-15  counterparts;  for  two  of 
the  warranted  LRUs,  the  reverse  was  true.  And  there  was  no 
statistical  difference  between  the  MFHBFs  of  the  F-16  and 
F-15  HUD  pilot  displays.  This  analysis  concluded,  as  did  the 
ARINC  interim  evaluation  of  the  F-16  RIW  (29:2-12),  that  the 
F-16  radar  transmitter  and  HUD  electronic  unit,  the  two  F-16 
LRUs  covered  by  a  MTBF  guarantee,  exhibited  significantly 
greater  MFHBFs  than  their  counterparts  in  the  F-15.  Two  sig¬ 
nificant  differences  were  noted,  however,  between  the  results 
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of  this  research  and  that  of  the  ARINC  study.  The  ARINC 
study  concluded  that  the  radar  computer  and  the  inertial  nav¬ 
igation  unit  exhibited  better  reliability  in  the  F-15.  How¬ 
ever,  this  research  has  shown  just  the  opposite,  that  these 
two  LRUs  exhibited  significantly  better  third  year  relia¬ 
bility  in  the  F-16. 

The  normalizing  process  to  account  for  differences  in 
system  complexity  tended  to  "favor"  the  F-15  LRUs,  since  all 
but  two  of  the  F-15  LRUs  had  a  greater  number  of  SRUs  per  LRU 
than  those  of  the  F-16.  In  the  two  cases  where  the  F-16  had 
the  "advantage",  the  radar  receiver  and  the  inertial  naviga¬ 
tion  unit,  the  MFHBFs  of  these  two  F-16  LRUs  were  greater 
than  those  of  their  F-15  counterparts  even  before  the  F-15 
MFHBFs  were  adjusted  by  the  "complexity  factor."  While  not 
part  of  this  research  design,  it  was  noted  that  had  the  F-15 
MFHBFs  not  been  normalized,  and  only  the  raw  MFHBFs  compared, 
the  results  would  have  changed  very  little.  Six  out  of  the 
nine  warranted  F-16  LRUs  would  still  have  shown  a  signifi¬ 
cantly  greater  reliability,  while  three  would  have  exhibited 
no  difference  in  reliability.  None  of  the  F-15  LRUs  would 
have  had  a  significantly  greater  MFHBF. 

Second  Research  Question 

To  review,  the  second  research  question  was  -  As  com¬ 
pared  with  functionally  similar  non-warranted  LRUs  in  both 
the  F-15  and  F-16,  did  the  nine  F-16  warranted  LRUs  exhibit  a 


significantly  higher  reliability  growth  during  the  first  four 
years  of  operation? 

The  second  research  question  was  really  comprised  of  two 
parts.  To  answer  the  first  part,  the  observed  reliability 
growth  over  a  four  year  period  of  each  of  the  nine  warranted 
F-1  6  LRUs  was  compared  to  the  reliability  growth  of  its 
equivalent  F-1 5  LRU(s).  To  answer  the  second  part  of  the 
question,  the  observed  reliability  growth  of  the  entire  F-1 6 
RIW  "system"  was  compared  to  the  growth  of  the  entire  F-1 6 
Non-RIW  "system",  where  the  monthly  MFHBFs  of  the  RIW  "sys¬ 
tem"  was  based  on  the  total  number  of  failures  reported  for 
all  the  nine  warranted  LRUs  and  the  monthly  MFHBFs  of  the 
Non-RIW  "system"  was  based  on  the  total  failures  of  all  nine 
non-warranted  F-1 6  LRUs. 

The  growth  parameter,  /3  ,  of  the  AMSAA  Reliability  Growth 
Model  was  calculated  for  each  of  the  nine  F-1 5  and  F-1 6  LRUs 
and  also  for  both  the  F-1 6  RIW  and  Non-RIW  "systems."  The 
growth  parameters,  which  are  measures  of  reliability  growth, 
were  calculated  to  four  decimal  places  using  the  methodology 
described  in  Chapter  II.  Monthly  failure  and  flight  time 
data  for  the  entire  respective  four  year  time  periods  were 
used  in  the  calculation  of  the  growth  parameters.  Thus,  48 
iterations  of  Eq  (10)  were  required  just  to  get  one  estima¬ 
tion  for  a  single  iteration  of  the  secant  formula,  Eq  (11). 
And  an  average  of  seven  iterations  of  the  secant  formula  were 
required  to  converge  to  obtain  an  estimate  of  (3  to  four 
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TABLE  IX 


RELIABILITY  GROWTH  PARAMETERS,  j3,  AND  TOTAL  FAILURES 
OF  F-16  LRUs  AND  EQUIVALENT  F-15  LRUs 


F-1 6  LRUs 


F-15  LRUs 


LRU  Description 

Number  of 
Failures 

mm 

Number  of 
Failures 

0 

15 

Flight  Control  Comp. 

609 

.8598 

383 

.8529 

Radar  Antenna 

1207 

.7712 

1013 

.8325 

Radar  Receiver 

1177 

.7964 

855 

.9063 

Radar  Transmitter 

803 

.7084 

1  344 

.8553 

Radar  Signal  Proc. 

455 

.7234 

959 

.7063 

Radar  Computer 

593 

.6229 

1250 

.7647 

HUD  Pilot  Display 

595 

.8483 

718 

.8129 

HUD  Electronics 

338 

.8850 

534 

.6456 

Inertial  Nav.  Unit 

1375 

.8462 

1392 

.6857 

RIW  Combined  Total 

71  52 

.7802 

Non-RIW  Combined  Total 

3416 

.8487 

i 


Source:  (54) 

Note:  The  total  failures  are  based  on  the  following  time 
periods:  F-16  -  January  1979  to  December  1982;  F-15  - 
January  1975  to  December  1978. 


decimal  places.  Table  IX  contains  the  results  of  the  AMSAA 
Model  growth  parameter  calculations  along  with  the  total 
number  of  failures  for  each  of  the  LRUs  for  the  entire  re¬ 
spective  four  year  period.  Listed  at  the  bottom  of  Table  IX 


are  the  results  of  the  calculations  for  the  comparison  of  the 
F-16  RIW  versus  the  Non-RIW  "systems." 

A  statistical  F  test  was  used  to  determine  whether  the 
reliability  growths,  as  measured  by  /3,  of  the  warranted  F-16 
LRUs  were  significantly  greater  than  the  growths  of  the 
equivalent  F-1 5  LRUs,  at  a  five  percent  alpha  level.  An 
identical  F  test  was  used  to  determine  whether  the  relia¬ 
bility  growth  of  the  RIW  "system"  was  significantly  greater 
than  that  of  the  Non-RIW  "system."  The  null  hypothesis  for 
the  first  part  of  the  question  stated  there  was  no  discern¬ 
ible  difference;  the  alternate  hypothesis  stated  that  the 
reliability  growth  of  the  F-16  LRU  was  greater  than  that  of 
its  F-1 5  counterpart  or  in  this  case  the  growth  parameter, 

/3  ,  of  the  F-16  LRU  was  less  than  the  growth  parameter,  0  , 

of  the  F-1 5  LRU: 

V  *16  =  *15 

V  *16  <  *15 

The  null  hypothesis  for  the  second  part  of  the  question  also 
stated  there  was  no  difference,  while  the  alternate  hypoth¬ 
esis  stated  that  the  reliability  growth  of  the  RIW  "system" 
was  greater  than  that  of  the  Non-RIW  "system": 

H0  *  *RIld  *N0NRI hi 

H  :  0  <  (3 

a  RIW  NONRIW 


TABLE  X 


SUMMARY  OF  F  TEST  RESULTS 

F-1 6  RIW  LRU  VERSUS  F-1 5  LRU  RELIABILITY  GROWTH  COMPARISON 


LRU  Description 

Fc 

F  .05 

Decision 

Flight  Control  Comp. 

.99 

1  .11 

test  further 

Radar  Antenna 

1  .08 

1  .07 

reject 

Radar  Receiver 

1.14 

1  .08 

rej  ect 

Radar  Transmitter 

1  .21 

1  .08 

reject 

Radar  Signal  Proc. 

.98 

1.10 

test  further 

Radar  Computer 

1.23 

1  .09 

reject 

HUD  Pilot  Display 

.96 

1.10 

test  further 

HUD  Electronics 

.73 

1  .12 

test  further 

Inertial  Nav.  Unit 

.81 

1  .06 

test  further 

RIW  vs.  Non-RIW  system 

1  .09 

1.03 

reject 

Table  X  contains  the  summary  of  the  F  test  results.  Again, 
the  technique  explained  in  Appendix  B  was  used  to  approximate 

the  critical  F  values  listed  in  Table  X. 

•05 

The  F  test  results  indicated  that  the  null  hypothesis  of 
equal  reliability  growths  can  be  rejected  in  only  four  out  of 
the  nine  LRU  comparisons.  It  was  therefore  concluded  that 
the  radar  antenna,  receiver,  transmitter,  and  computer  all 
exhibited  significantly  higher  reliability  growths  than  their 
counterpart  LRUs  in  the  F-1 5. 


Again,  a  corollary  hypothesis  test  was  used  to  compare 


the  reliability  growths  of  those  pairs  of  LRUs  that  could  not 
be  rejected  under  the  initial  hypothesis  test.  The  purpose 
of  the  corollary  test  was  to  determine  if  the  observed  relia¬ 
bility  growths  cf  the  F-15  LRUs  were,  in  fact,  greater  than 
those  of  the  warranted  F-16  LRUs.  This  second  corollary  null 
hypothesis  stated  that  there  was  no  discernible  difference 
and  the  alternate  hypothesis  stated  that  the  reliability 
growth  of  the  F-15  LRU  was  greater  than  that  of  its  F-16 
counterpart  or  in  this  case  the  growth  parameter,  /315  ,  of  the 

F-15  LRU  was  less  than  the  growth  parameter,  8  ,  of  the  war- 

1 6 

ranted  F-16  LRU: 


H  : 
o 

H  : 
a 


P 

P 


16 

16 


Table  XI  summarizes  the  results  of  the  F  tests  for  this 
corollary  hypothesis.  The  results  indicated  that  the  null 
hypothesis  can  be  rejected  for  two  of  the  five  corollary  LRU 
comparisons.  It  was  thus  concluded  that  the  F-15  HUD  elec¬ 
tronic  and  inertial  navigation  units  exhibited  significantly 
greater  reliability  growth  over  the  four  year  period  than 
their  warranted  F-16  counterparts.  Furthermore,  for  three  of 
the  LRUs,  the  flight  control  computer,  the  radar  signal 
processor,  and  the  HUD  pilot  display,  this  research  concluded 
there  was  no  statistical  difference  in  the  reliability 
growths  between  those  in  the  F-15  and  those  in  the  F-16. 


TABLE  XI 


SUMMARY  OF  F  TEST  RESULTS  -  F-T6  RIW  LRU  VERSUS  F-15  LRU 
COROLLARY  RELIABILITY  GROWTH  COMPARISON 


LRU  Description 

Fc 

Decision 

Flight  Control  Comp. 

1  .01 

i.ii 

do  not  reject 

Radar  Signal  Proc. 

1  .02 

1  .10 

do  not  reject 

HUD  Pilot  Display 

1  .04 

1  .10 

do  not  reject 

HUD  Electronics 

1.37 

1.12 

reject 

Inertial  Nav.  Unit 

1  .23 

1  .06 

rej  ect 

The  F  test  results  for  the  second  part  of  research 
question  two  indicated  that  the  null  hypothesis  of  equal 
reliability  growths  for  the  RIW  and  Non-RIW  "systems"  can  be 
rejected.  It  was  therefore  concluded  that  the  "system"  of 
warranted  F-16  LRUs  as  a  group  exhibited  a  significantly 
higher  reliability  growth  than  the  entire  "system"  of  non- 
warranted  F-16  LRUs. 

To  summarize  the  results  of  research  question  two,  four 
of  the  nine  warranted  F-16  LRUs  exhibited  significantly 
greater  reliability  growths  than  their  F-15  counterparts  over 
the  respective  four  year  focal  time  periods.  On  the  other 
hand,  two  non-warranted  F-15  LRUs  displayed  greater  growths 
than  their  warranted  counterparts  in  the  F-16.  There  was  no 
statistical  difference  between  the  reliability  growths  ob¬ 
served  for  three  of  the  LRU  pairs.  And  finally,  the  relia¬ 
bility  growth  of  the  entire  group  of  nine  warranted  LRUs  was 
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shown  to  be  significantly  greater  than  that  of  the  entire 
group  of  nine  non-warranted  F-16  LRUs. 

There  were  no  striking  differences  between  the  results 
of  this  study,  which  covered  a  four  year  time  span,  and  those 
of  the  ARINC  interim  evaluation,  which  covered  a  two  year 
period.  Their  research  also  concluded  that  four  warranted 
F-16  LRUs  displayed  higher  growth  rates.  They  did  show,  how¬ 
ever,  that  there  was  no  statistical  difference  between  the 
reliability  growths  of  the  other  five  LRU  pairs;  they  ob¬ 
served  that  none  of  the  F-1 5  LRUs  exhibited  a  statistically 
higher  growth  rate.  The  ARINC  report  concluded  there  was  no 
difference  between  the  growths  of  the  RIW  and  Non-RIW  "sys¬ 
tems,"  which  also  contradicts  the  findings  of  this  study. 
(29:2-19) 

The  results  of  this  study  suggest  that  one  subcontrac¬ 
tor,  Westinghouse ,  the  manufacturer  of  the  fire  control 
radar,  was  responsible  for  fielding  a  system  that  not  only 
exhibited  a  greater  MFHBF  than  its  F-1 5  counterpart,  but  also 
surpassed  the  F-1 5  system  in  reliability  growth  over  equiva¬ 
lent  four  year  periods.  All  four  of  the  warranted  F-16  LRUs 
displaying  significantly  greater  reliability  growths  were 
components  of  the  Westinghouse  radar.  No  other  similar  con¬ 
clusions  could  be  drawn  about  the  other  three  subcontractors 
involved  in  the  F-16  RIW. 
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IV.  Conclusions  and  Recommendations 


Summary 

To  successfully  counter  the  perceived  threat  to  our 
national  security,  the  Department  of  Defense  has  acquired  a 
succession  of  weapon  systems,  each  of  which  is  more  complex 
and  more  costly  than  its  predecessor.  It  is  of  utmost  im¬ 
portance  that  military  procurement  agencies  use  the  most  cost 
effective  "tools"  available  to  them  in  the  acquisition  of 
these  sophisticated  systems.  The  reliability  improvement 
warranty  (RIW)  is  one  such  "tool"  which  has  the  potential  for 
reducing  a  system's  total  life  cycle  costs.  Unfortunately, 
very  little  research  to  determine  the  effectiveness  of  the 
RIW  in  DOD  acquisition  has  surfaced.  Therefore,  the  objec¬ 
tive  of  this  thesis  was  to  investigate  what  effects,  if  any, 
a  RIW  had  on  the  reliability  of  warranted  avionics  equipment. 
Specifically,  this  study  was  designed  to  answer  the  question 
of  whether  or  not  a  warranted  system  was  significantly  more 
reliable  than  it  would  have  been  without  the  warranty.  To 
accomplish  this,  the  most  comprehensive  DOD  application  of 
RIW  to  date,  the  F-16  RIW,  was  investigated. 

The  thesis  objective  was  attained  by  answering  the  two 
research  questions,  which  formed  the  basic  framework  for  the 
research  effort: 

1.  As  compared  with  the  reliabilities  (MTBFs)  of  nine 
functionally  similar  non-warranted  line  replaceable  units 


(LRUs)  in  the  F-15,  are  the  reliabilities  of  the  nine  F-16 
warranted  LRUs  significantly  higher? 


2.  As  compared  with  functionally  similar  non-warranted 
LRUs  in  both  the  F-15  and  F-16,  did  the  nine  F-16  warranted 
LRUs  exhibit  a  significantly  higher  reliability  growth  during 
the  first  four  years  of  operation? 

The  information  used  in  the  analysis  was  obtained  from 
the  Air  Force  Maintenance  Data  Collection  System,  or  more 
specifically,  the  D056  system.  The  data  consisted  of  monthly 
failures  and  flight  hours  for  each  of  the  LRUs  included  in 
the  study.  To  answer  the  first  question,  the  entire  third 
year  after  the  start  of  production  was  used  as  a  basis  (1977 
for  the  F-15  and  1981  for  the  F-16).  Comparative  four  year 
periods  were  used  as  a  basis  for  the  second  question  (1975  to 
1978  for  the  F-15  and  1979  to  1982  for  the  F-16).  Chapter  II 
contains  a  complete  explanation  of  the  research  methodology 
employed  to  analyze  the  data  and  answer  the  two  research 
questions.  The  reader  should  review  the  assumptions  and 
limitations  presented  in  that  chapter,  before  accepting  the 
findings  and  conclusions  of  this  thesis.  The  conclusions 
apply  only  to  the  nine  LRUs  covered  by  the  F-16  warranty;  no 
attempt  was  made  to  generalize  the  results  to  equipment 
covered  by  other  reliability  improvement  warranties. 

Findings 

Research  Question  One.  As  a  result  of  the  statistical 
comparisons,  the  answer  to  the  first  research  question  was  an 


affirmative.  The  warranted  F-16  LRUs,  in  general,  demon¬ 
strated  statistically  greater  reliability,  as  measured  by 
mean  flight  hours  between  failures  (MFHBFs),  than  the  non- 
warranted  F-15  LRUs.  More  specifically,  of  the  nine  war¬ 
ranted  F-16  LRUs,  six  exhibited  statistically  greater  third 
year  MFHBFs  than  their  F-15  counterparts,  two  exhibited  sta¬ 
tistically  less  reliability  than  the  equivalent  F-15  LRUs, 
and  there  was  no  difference  in  the  reliability  of  one  of  the 
LRU  pairs. 

Research  Question  Two.  The  results  of  the  statistical 
tests  used  to  answer  the  second  question  were  somewhat  less 
conclusive  than  those  used  to  answer  the  first.  Four  of  the 
nine  warranted  F-16  LRUs  exhibited  statistically  greater 
reliability  growths  than  their  F-15  counterparts  over  the 
respective  four  year  focal  time  periods.  Two  displayed 
statistically  less  growth  than  their  F-15  counterparts,  and 
there  was  no  statistical  difference  between  the  reliability 
growths  observed  for  the  remaining  three  LRU  pairs.  The 
second  part  of  question  two  compared  the  reliability  growth 
of  the  combined  total  of  the  entire  group  of  nine  warranted 
F-16  LRUs  (defined  as  the  RIW  "system”)  with  that  of  the 
combined  total  of  a  group  of  nine  non-warranted  F-16  avionics 
LRUs  (Non-RIW  "system").  The  reliability  growth  of  the  RIW 
"system"  was  shown  to  be  statistically  greater  than  that  of 
the  Non-RIW  "system."  The  evidence  obtained  in  this  study  to 


answer  the  second  research  question  was  not  convincing  enough 
to  give  it  either  an  affirmative  or  a  negative  answer. 

Conclusions 

While  not  overwhelming,  there  was  a  convincing  degree  of 
evidence  that  indicated  the  reliability  of  the  warranted  F-16 
LRUs  was  greater  than  the  reliability  of  functionally  similar 
non-warranted  F-15  LRUs.  This  study  also  concluded  that  the 
reliability  growth  rate  of  the  warranted  LRUs,  taken  as  a 
group,  was  significantly  greater  than  that  of  the  non-war- 
ranted  F-16  LRUs,  taken  as  a  group. 

The  next  logical  conclusion  to  make,  then,  would  have 
been  that  the  F-16  RIW  program  was  the  significant  causal 
factor  for  this  observed  increase  in  reliability.  Since  this 
study  was  not  specifically  designed  to  make  that  inference, 
technically,  that  conclusion  can  not  be  drawn  from  the  re¬ 
sults.  Harrison  identified  four  conditions,  however,  that 
when  met  and  the  resultant  action  taken,  would  reasonably 
attribute  an  observed  improvement  in  reliability  to  the  RIW. 
The  conditions  are  the  following  (29:2-1): 

1 .  A  desirable  action  is  identified. 

2.  The  action  has  clear  economic  advantages  (to  the 
contractor,  assuming  the  contractor  is  motivated  by  profit.) 

3.  Risks  are  favorable  with  respect  to  economic 
returns . 

4.  There  is  opportunity  to  implement  the  action. 
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The  first  three,  the  contractor  implicitly  control;  the  last 
condition,  the  government  controls,  through  its  approval 
authority  (29:2-1).  There  was  no  reason  to  believe  the  above 
four  conditions  were  not  present  during  the  four  year  war¬ 
ranty  period.  That  being  the  case,  this  study  concluded  that 
the  observed  increased  reliability  and  reliablity  growth  rate 
of  the  warranted  F-16  LRUs,  was  due,  at  least  in  part,  to  the 
F-16  reliability  improvement  warranty. 

Recommendations  for  Future  Research 

Probably  the  area  of  recommended  research  that  would 
provide  the  most  significant  impact,  at  least  in  relation  to 
this  thesis,  would  be  a  complete  economic  analysis  of  the 
F-16  RIW  program,  now  that  the  program  is  concluded.  Accord¬ 
ing  to  the  F-16  Program  Office  personnel  responsible  for 
closing  out  the  program,  no  evaluation  of  the  RIW  cost  im¬ 
pacts  was  ever  undertaken.  An  ARINC  Research  Corporation 
interim  report  did  provide  an  extensive  economic  evaluation 
of  the  F-16  RIW;  however,  that  evaluation  was  based  on  data 
from  the  first  two  years  of  the  warranty,  which  really 
represented  less  than  one  fourth  of  the  actual  accrued  flying 
time  for  the  entire  warranty  period.  Since  this  has  been  the 
most  extensive  DOD  use  of  a  warranty  to  date,  an  exhaustive 
economic  evaluation  of  this  program  would  certainly  be  in 
order . 

Additionally,  parallel  research  efforts  should  be  per¬ 
formed  to  evaluate  the  effectiveness,  in  terms  of  reliability 
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APPENDIX  B:  Critical  F  Value  Approximation 


Since  percentage  points  of  the  F  distribution  tables 
(F  tables)  only  tabulate  critical  F  values  for  degrees  of 
freedom  up  to  120,  the  following  approximation  formula,  from 
Applied  Statistics  -  A  Handbook  of  Techniques ,  by  Lothar 
Sachs,  was  used  (55:151): 


log  F 


(0.4343)  (z) 


/  2(W 

/  y,  y2 


(13) 


where  z  is  the  standard  normal  value  for  the  chosen  signifi¬ 
cance  level  of  a  one-sided  test  and  Y  and  y  are  the  numer- 

1  2 

ator  and  denominator  degrees  of  freedom,  respectively. 

For  example,  the  approximate  value  of  F  (348,  248;  .05 
is  given  by 


log  F 


(0.4343)  (1.645) 


/ 2(348+248) 
/  (348) (248) 


0.0840 


F 


10 


0840 


1.213 
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